- Dec 25, 2019
-
-
Todd Gamblin authored
Remove a few remaining mentions of the pyqver package, which was removed in #14289.
-
- Dec 24, 2019
-
-
Massimiliano Culpo authored
This commit removes the `python_version.py` unit test module and the vendored dependencies `pyqver2.py` and `pyqver3.py`. It substitutes them with an equivalent check done using `vermin` that is run as a separate workflow via Github Actions. This allows us to delete 2 vendored dependencies that are unmaintained and substitutes them with a maintained tool. Also, updates the list of vendored dependencies.
-
t-karatsu authored
-
Matthias Diener authored
-
Matthias Diener authored
-
Matthias Diener authored
-
健美猞猁 authored
-
Todd Gamblin authored
-
Todd Gamblin authored
-
Massimiliano Culpo authored
Before this commit we used to run the entire unit test suite in the presence of a failure. Since we currently rely a lot on the state of the filesystem etc. the end report was most of the time showing spurious failures that were a consequence of the first failing test. This PR makes unit tests exit at the first failing test Also, pin codecov at v4.5.4 (last one supporting Python 2.6)
-
Todd Gamblin authored
`ViewDescriptor.regenerate()` calls `get_all_specs()`, which reads `spec.yaml` files, which is slow. It's fine to do this once, but `view.remove_specs()` *also* calls it immediately afterwards. - [x] Pass the result of `get_all_specs()` as an optional parameter to `view.remove_specs()` to avoid reading `spec.yaml` files twice.
-
Todd Gamblin authored
`ViewDescriptor.regenerate()` was copying specs and stripping build dependencies, which clears `_hash` and other cached fields on concrete specs, which causes a bunch of YAML hashes to be recomputed. - [x] Preserve the `_hash` and `_normal` fields on stripped specs, as these will be unchanged.
-
Todd Gamblin authored
`os.path.exists()` will report False if the target of a symlink doesn't exist, so we can avoid a costly call to realpath here.
-
Todd Gamblin authored
`spack install` previously concretized, writes the entire environment out, regenerated views, then wrote and regenerated views again. Regenerating views is slow, so ensure that we only do that once. - [x] add an option to env.write() to skip view regeneration - [x] add a note on whether regenerate_views() shouldn't just be a separate operation -- not clear if we want to keep it as part of write to ensure consistency, or take it out to avoid performance issues.
-
Todd Gamblin authored
Environments need to read the DB a lot when installing all specs. - [x] Put a read transaction around `install_all()` and `install()` to avoid repeated locking
-
Todd Gamblin authored
Our `LockTransaction` class was reading overly aggressively. In cases like this: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... ``` The `ReadTransaction` on line 1 would read in the DB, but the WriteTransaction on line 2 would read in the DB *again*, even though we had a read lock the whole time. `WriteTransaction`s were only considering nested writes to decide when to read, but they didn't know when we already had a read lock. - [x] `Lock.acquire_write()` return `False` in cases where we already had a read lock.
-
Todd Gamblin authored
If a write transaction was nested inside a read transaction, it would not write properly on release, e.g., in a sequence like this, inside our `LockTransaction` class: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... 4 with spack.store.db.read_transaction(): ... ``` The WriteTransaction on line 2 had no way of knowing that its `__exit__()` call was the last *write* in the nesting, and it would skip calling its write function. The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know how to write, and the file would never be written. The DB would be correct in memory, but the `ReadTransaction` on line 4 would re-read the whole DB assuming that other processes may have modified it. Since the DB was never written, we got stale data. - [x] Make `Lock.release_write()` return `True` whenever we release the *last write* in a nest.
-
Todd Gamblin authored
Lock transactions were actually writing *after* the lock was released. The code was looking at the result of `release_write()` before writing, then writing based on whether the lock was released. This is pretty obviously wrong. - [x] Refactor `Lock` so that a release function can be passed to the `Lock` and called *only* when a lock is really released. - [x] Refactor `LockTransaction` classes to use the release function instead of checking the return value of `release_read()` / `release_write()`
-
Todd Gamblin authored
`ViewDescriptor.regenerate()` checks repeatedly whether packages are installed and also does a lot of DB queries. Put a read transaction around the whole thing to avoid repeatedly locking and unlocking the DB.
-
Todd Gamblin authored
`Environment.added_specs()` has a loop around calls to `Package.installed()`, which can result in repeated DB queries. Optimize this with a read transaction in `Environment`.
-
Todd Gamblin authored
`spack spec -I` queries the database for installation status and should use a read transaction around calls to `Spec.tree()`.
-
Todd Gamblin authored
Checks for deprecated specs were repeatedly taking out read locks on the database, which can be very slow. - [x] put a read transaction around the deprecation check
-
Todd Gamblin authored
`get_platform()` is pretty expensive and can be called many times in a spack invocation. - [x] memoize `get_platform()`
-
Sajid Ali authored
-
Peter Josef Scheibel authored
-
Peter Josef Scheibel authored
BundlePackages use a noop fetch strategy. The mirror logic was assuming that the fetcher had a resource to cach after performing a fetch. This adds a special check to skip caching if the stage is associated with a BundleFetchStrategy. Note that this should allow caching resources associated with BundlePackages.
-
Peter Josef Scheibel authored
When updating a mirror, Spack was re-retrieving all patches (since the fetch logic for patches is separate). This updates the patch logic to allow the mirror logic to avoid this.
-
Peter Josef Scheibel authored
Since cache_mirror does the fetch itself, it also needs to do the checksum itself if it wants to verify that the source stored in the mirror is valid. Note that this isn't strictly required because fetching (including from mirrors) always separately verifies the checksum.
-
Peter Josef Scheibel authored
The targets for the cosmetic paths in mirrrors were being calculated incorrectly as of fb3a3ba9: the symlinks used relative paths as targets, and the relative path was computed relative to the wrong directory.
-
Peter Josef Scheibel authored
When creating a cosmetic symlink for a resource in a mirror, remove it if it already exists. The symlink is removed in case the logic to create the symlink has changed.
-
Paul Ferrell authored
* Some packages (e.g. mpfr at the time of this patch) can have patches with the same name but different contents (which apply to different versions of the package). This appends part of the patch hash to the cache file name to avoid conflicts. * Some exceptions which occur during fetching are not a subclass of SpackError and therefore do not have a 'message' attribute. This updates the logic for mirroring a single spec (add_single_spec) to produce an appropriate error message in that case (where before it failed with an AttributeError) * In various circumstances, a mirror can contain the universal storage path but not a cosmetic symlink; in this case it would not generate a symlink. Now "spack mirror create" will create a symlink for any package that doesn't have one.
-
Todd Gamblin authored
`ViewDescriptor.regenerate()` calls `get_all_specs()`, which reads `spec.yaml` files, which is slow. It's fine to do this once, but `view.remove_specs()` *also* calls it immediately afterwards. - [x] Pass the result of `get_all_specs()` as an optional parameter to `view.remove_specs()` to avoid reading `spec.yaml` files twice.
-
Todd Gamblin authored
`ViewDescriptor.regenerate()` was copying specs and stripping build dependencies, which clears `_hash` and other cached fields on concrete specs, which causes a bunch of YAML hashes to be recomputed. - [x] Preserve the `_hash` and `_normal` fields on stripped specs, as these will be unchanged.
-
Todd Gamblin authored
`os.path.exists()` will report False if the target of a symlink doesn't exist, so we can avoid a costly call to realpath here.
-
Todd Gamblin authored
`spack install` previously concretized, writes the entire environment out, regenerated views, then wrote and regenerated views again. Regenerating views is slow, so ensure that we only do that once. - [x] add an option to env.write() to skip view regeneration - [x] add a note on whether regenerate_views() shouldn't just be a separate operation -- not clear if we want to keep it as part of write to ensure consistency, or take it out to avoid performance issues.
-
Todd Gamblin authored
Environments need to read the DB a lot when installing all specs. - [x] Put a read transaction around `install_all()` and `install()` to avoid repeated locking
-
Todd Gamblin authored
Our `LockTransaction` class was reading overly aggressively. In cases like this: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... ``` The `ReadTransaction` on line 1 would read in the DB, but the WriteTransaction on line 2 would read in the DB *again*, even though we had a read lock the whole time. `WriteTransaction`s were only considering nested writes to decide when to read, but they didn't know when we already had a read lock. - [x] `Lock.acquire_write()` return `False` in cases where we already had a read lock.
-
Todd Gamblin authored
If a write transaction was nested inside a read transaction, it would not write properly on release, e.g., in a sequence like this, inside our `LockTransaction` class: ``` 1 with spack.store.db.read_transaction(): 2 with spack.store.db.write_transaction(): 3 ... 4 with spack.store.db.read_transaction(): ... ``` The WriteTransaction on line 2 had no way of knowing that its `__exit__()` call was the last *write* in the nesting, and it would skip calling its write function. The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know how to write, and the file would never be written. The DB would be correct in memory, but the `ReadTransaction` on line 4 would re-read the whole DB assuming that other processes may have modified it. Since the DB was never written, we got stale data. - [x] Make `Lock.release_write()` return `True` whenever we release the *last write* in a nest.
-
Todd Gamblin authored
Lock transactions were actually writing *after* the lock was released. The code was looking at the result of `release_write()` before writing, then writing based on whether the lock was released. This is pretty obviously wrong. - [x] Refactor `Lock` so that a release function can be passed to the `Lock` and called *only* when a lock is really released. - [x] Refactor `LockTransaction` classes to use the release function instead of checking the return value of `release_read()` / `release_write()`
-