Skip to content
Snippets Groups Projects
  1. Dec 25, 2019
  2. Dec 24, 2019
    • Massimiliano Culpo's avatar
      tests: check min required python version with vermin (#14289) · d333e147
      Massimiliano Culpo authored
      This commit removes the `python_version.py` unit test module
      and the vendored dependencies `pyqver2.py` and `pyqver3.py`.
      It substitutes them with an equivalent check done using
      `vermin` that is run as a separate workflow via Github Actions.
      
      This allows us to delete 2 vendored dependencies that are unmaintained
      and substitutes them with a maintained tool.
      
      Also, updates the list of vendored dependencies.
      d333e147
    • t-karatsu's avatar
      a64fx: fix typo in GCC flags (#14286) · 1e2c9d96
      t-karatsu authored
      1e2c9d96
    • Matthias Diener's avatar
      intel-pin: add version 3.11 (#14292) · 6a084a82
      Matthias Diener authored
      6a084a82
    • Matthias Diener's avatar
      cmake: add version 3.16.2 (#14291) · 37c1d2d0
      Matthias Diener authored
      37c1d2d0
    • Matthias Diener's avatar
      scons: add version 3.1.2 (#14290) · ee8bc0df
      Matthias Diener authored
      ee8bc0df
    • 健美猞猁's avatar
      Bump nektar to 5.0.0. (#14288) · 748c5529
      健美猞猁 authored
      748c5529
    • Todd Gamblin's avatar
      Merge branch 'releases/v0.13' into develop · 7652d1a4
      Todd Gamblin authored
      Unverified
      7652d1a4
    • Todd Gamblin's avatar
    • Todd Gamblin's avatar
      version bump: 0.13.3 · 231e2377
      Todd Gamblin authored
      Unverified
      231e2377
    • Massimiliano Culpo's avatar
      Travis exits at the first failing test, pin codecov at v4.5.4 (#14179) · 8616a264
      Massimiliano Culpo authored
      Before this commit we used to run the entire unit test suite
      in the presence of a failure. Since we currently rely a lot
      on the state of the filesystem etc. the end report was most
      of the time showing spurious failures that were a consequence
      of the first failing test.
      
      This PR makes unit tests exit at the first failing test
      
      Also, pin codecov at v4.5.4 (last one supporting Python 2.6)
      Unverified
      8616a264
    • Todd Gamblin's avatar
      performance: dont' read `spec.yaml` files twice in view regeneration · e22d3250
      Todd Gamblin authored
      `ViewDescriptor.regenerate()` calls `get_all_specs()`, which reads
      `spec.yaml` files, which is slow.  It's fine to do this once, but
      `view.remove_specs()` *also* calls it immediately afterwards.
      
      - [x] Pass the result of `get_all_specs()` as an optional parameter to
        `view.remove_specs()` to avoid reading `spec.yaml` files twice.
      Unverified
      e22d3250
    • Todd Gamblin's avatar
      performance: don't recompute hashes when regenerating environments · e3939b0c
      Todd Gamblin authored
      `ViewDescriptor.regenerate()` was copying specs and stripping build
      dependencies, which clears `_hash` and other cached fields on concrete
      specs, which causes a bunch of YAML hashes to be recomputed.
      
      - [x] Preserve the `_hash` and `_normal` fields on stripped specs, as
        these will be unchanged.
      Unverified
      e3939b0c
    • Todd Gamblin's avatar
      performance: reduce system calls required for remove_dead_links · f0136873
      Todd Gamblin authored
      `os.path.exists()` will report False if the target of a symlink doesn't
      exist, so we can avoid a costly call to realpath here.
      Unverified
      f0136873
    • Todd Gamblin's avatar
      performance: only regenerate env views once in `spack install` · 79ddf6cf
      Todd Gamblin authored
      `spack install` previously concretized, writes the entire environment
      out, regenerated views, then wrote and regenerated views
      again. Regenerating views is slow, so ensure that we only do that once.
      
      - [x] add an option to env.write() to skip view regeneration
      
      - [x] add a note on whether regenerate_views() shouldn't just be a
        separate operation -- not clear if we want to keep it as part of write
        to ensure consistency, or take it out to avoid performance issues.
      Unverified
      79ddf6cf
    • Todd Gamblin's avatar
      performance: add read transactions for `install_all()` and `install()` · be6d7db2
      Todd Gamblin authored
      Environments need to read the DB a lot when installing all specs.
      
      - [x] Put a read transaction around `install_all()` and `install()`
        to avoid repeated locking
      Unverified
      be6d7db2
    • Todd Gamblin's avatar
      lock transactions: avoid redundant reading in write transactions · d87ededd
      Todd Gamblin authored
      Our `LockTransaction` class was reading overly aggressively.  In cases
      like this:
      
      ```
      1  with spack.store.db.read_transaction():
      2    with spack.store.db.write_transaction():
      3      ...
      ```
      
      The `ReadTransaction` on line 1 would read in the DB, but the
      WriteTransaction on line 2 would read in the DB *again*, even though we
      had a read lock the whole time.  `WriteTransaction`s were only
      considering nested writes to decide when to read, but they didn't know
      when we already had a read lock.
      
      - [x] `Lock.acquire_write()` return `False` in cases where we already had
             a read lock.
      Unverified
      d87ededd
    • Todd Gamblin's avatar
      lock transactions: ensure that nested write transactions write · b3a5f2e3
      Todd Gamblin authored
      If a write transaction was nested inside a read transaction, it would not
      write properly on release, e.g., in a sequence like this, inside our
      `LockTransaction` class:
      
      ```
      1  with spack.store.db.read_transaction():
      2    with spack.store.db.write_transaction():
      3      ...
      4  with spack.store.db.read_transaction():
         ...
      ```
      
      The WriteTransaction on line 2 had no way of knowing that its
      `__exit__()` call was the last *write* in the nesting, and it would skip
      calling its write function.
      
      The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know
      how to write, and the file would never be written.
      
      The DB would be correct in memory, but the `ReadTransaction` on line 4
      would re-read the whole DB assuming that other processes may have
      modified it.  Since the DB was never written, we got stale data.
      
      - [x] Make `Lock.release_write()` return `True` whenever we release the
            *last write* in a nest.
      Unverified
      b3a5f2e3
    • Todd Gamblin's avatar
      lock transactions: fix non-transactional writes · 98577e3a
      Todd Gamblin authored
      Lock transactions were actually writing *after* the lock was
      released. The code was looking at the result of `release_write()` before
      writing, then writing based on whether the lock was released.  This is
      pretty obviously wrong.
      
      - [x] Refactor `Lock` so that a release function can be passed to the
            `Lock` and called *only* when a lock is really released.
      
      - [x] Refactor `LockTransaction` classes to use the release function
        instead of checking the return value of `release_read()` / `release_write()`
      Unverified
      98577e3a
    • Todd Gamblin's avatar
      performance: avoid repeated DB locking on view generation · a85b9070
      Todd Gamblin authored
      `ViewDescriptor.regenerate()` checks repeatedly whether packages are
      installed and also does a lot of DB queries.  Put a read transaction
      around the whole thing to avoid repeatedly locking and unlocking the DB.
      Unverified
      a85b9070
    • Todd Gamblin's avatar
      performance: speed up `spack find` in environments · 91ea90c2
      Todd Gamblin authored
      `Environment.added_specs()` has a loop around calls to
      `Package.installed()`, which can result in repeated DB queries.  Optimize
      this with a read transaction in `Environment`.
      Unverified
      91ea90c2
    • Todd Gamblin's avatar
      performance: `spack spec` should use a read transacction with -I · 5bdba988
      Todd Gamblin authored
      `spack spec -I` queries the database for installation status and should
      use a read transaction around calls to `Spec.tree()`.
      Unverified
      5bdba988
    • Todd Gamblin's avatar
      concretization: improve performance by avoiding database locks · cbf85534
      Todd Gamblin authored
      Checks for deprecated specs were repeatedly taking out read locks on the
      database, which can be very slow.
      
      - [x] put a read transaction around the deprecation check
      Unverified
      cbf85534
    • Todd Gamblin's avatar
      performance: memoize spack.architecture.get_platform() · 48befd67
      Todd Gamblin authored
      `get_platform()` is pretty expensive and can be called many times in a
      spack invocation.
      
      - [x] memoize `get_platform()`
      Unverified
      48befd67
    • Sajid Ali's avatar
    • Peter Josef Scheibel's avatar
    • Peter Josef Scheibel's avatar
      Mirrors: skip attempts to fetch BundlePackages · 587c650b
      Peter Josef Scheibel authored
      BundlePackages use a noop fetch strategy. The mirror logic was assuming
      that the fetcher had a resource to cach after performing a fetch. This adds
      a special check to skip caching if the stage is associated with a
      BundleFetchStrategy. Note that this should allow caching resources
      associated with BundlePackages.
      Unverified
      587c650b
    • Peter Josef Scheibel's avatar
      Mirrors: avoid re-downloading patches · d7142862
      Peter Josef Scheibel authored
      When updating a mirror, Spack was re-retrieving all patches (since the
      fetch logic for patches is separate). This updates the patch logic to
      allow the mirror logic to avoid this.
      Unverified
      d7142862
    • Peter Josef Scheibel's avatar
      Mirrors: perform checksum of fetched sources · a69b3c85
      Peter Josef Scheibel authored
      Since cache_mirror does the fetch itself, it also needs to do the
      checksum itself if it wants to verify that the source stored in the
      mirror is valid. Note that this isn't strictly required because fetching
      (including from mirrors) always separately verifies the checksum.
      Unverified
      a69b3c85
    • Peter Josef Scheibel's avatar
      Mirrors: fix cosmetic symlink targets · 98b498c6
      Peter Josef Scheibel authored
      The targets for the cosmetic paths in mirrrors were being calculated
      incorrectly as of fb3a3ba9: the symlinks used relative paths as targets,
      and the relative path was computed relative to the wrong directory.
      Unverified
      98b498c6
    • Peter Josef Scheibel's avatar
      Allow repeated invocations of 'mirror create' · 64209dda
      Peter Josef Scheibel authored
      When creating a cosmetic symlink for a resource in a mirror, remove
      it if it already exists. The symlink is removed in case the logic to
      create the symlink has changed.
      Unverified
      64209dda
    • Paul Ferrell's avatar
      mirror bug fixes: symlinks, duplicate patch names, and exception handling (#13789) · c15e55c6
      Paul Ferrell authored
      * Some packages (e.g. mpfr at the time of this patch) can have patches
        with the same name but different contents (which apply to different
        versions of the package). This appends part of the patch hash to the
        cache file name to avoid conflicts.
      * Some exceptions which occur during fetching are not a subclass of
        SpackError and therefore do not have a 'message' attribute. This
        updates the logic for mirroring a single spec (add_single_spec)
        to produce an appropriate error message in that case (where before
        it failed with an AttributeError)
      * In various circumstances, a mirror can contain the universal storage
        path but not a cosmetic symlink; in this case it would not generate
        a symlink. Now "spack mirror create" will create a symlink for any
        package that doesn't have one.
      Unverified
      c15e55c6
    • Todd Gamblin's avatar
      performance: dont' read `spec.yaml` files twice in view regeneration · d7f2a328
      Todd Gamblin authored
      `ViewDescriptor.regenerate()` calls `get_all_specs()`, which reads
      `spec.yaml` files, which is slow.  It's fine to do this once, but
      `view.remove_specs()` *also* calls it immediately afterwards.
      
      - [x] Pass the result of `get_all_specs()` as an optional parameter to
        `view.remove_specs()` to avoid reading `spec.yaml` files twice.
      d7f2a328
    • Todd Gamblin's avatar
      performance: don't recompute hashes when regenerating environments · 78b84e4a
      Todd Gamblin authored
      `ViewDescriptor.regenerate()` was copying specs and stripping build
      dependencies, which clears `_hash` and other cached fields on concrete
      specs, which causes a bunch of YAML hashes to be recomputed.
      
      - [x] Preserve the `_hash` and `_normal` fields on stripped specs, as
        these will be unchanged.
      78b84e4a
    • Todd Gamblin's avatar
      performance: reduce system calls required for remove_dead_links · 9b90d7e8
      Todd Gamblin authored
      `os.path.exists()` will report False if the target of a symlink doesn't
      exist, so we can avoid a costly call to realpath here.
      9b90d7e8
    • Todd Gamblin's avatar
      performance: only regenerate env views once in `spack install` · c83e365c
      Todd Gamblin authored
      `spack install` previously concretized, writes the entire environment
      out, regenerated views, then wrote and regenerated views
      again. Regenerating views is slow, so ensure that we only do that once.
      
      - [x] add an option to env.write() to skip view regeneration
      
      - [x] add a note on whether regenerate_views() shouldn't just be a
        separate operation -- not clear if we want to keep it as part of write
        to ensure consistency, or take it out to avoid performance issues.
      c83e365c
    • Todd Gamblin's avatar
      performance: add read transactions for `install_all()` and `install()` · 0fb32800
      Todd Gamblin authored
      Environments need to read the DB a lot when installing all specs.
      
      - [x] Put a read transaction around `install_all()` and `install()`
        to avoid repeated locking
      0fb32800
    • Todd Gamblin's avatar
      lock transactions: avoid redundant reading in write transactions · 6c9467e8
      Todd Gamblin authored
      Our `LockTransaction` class was reading overly aggressively.  In cases
      like this:
      
      ```
      1  with spack.store.db.read_transaction():
      2    with spack.store.db.write_transaction():
      3      ...
      ```
      
      The `ReadTransaction` on line 1 would read in the DB, but the
      WriteTransaction on line 2 would read in the DB *again*, even though we
      had a read lock the whole time.  `WriteTransaction`s were only
      considering nested writes to decide when to read, but they didn't know
      when we already had a read lock.
      
      - [x] `Lock.acquire_write()` return `False` in cases where we already had
             a read lock.
      6c9467e8
    • Todd Gamblin's avatar
      lock transactions: ensure that nested write transactions write · bb517fdb
      Todd Gamblin authored
      If a write transaction was nested inside a read transaction, it would not
      write properly on release, e.g., in a sequence like this, inside our
      `LockTransaction` class:
      
      ```
      1  with spack.store.db.read_transaction():
      2    with spack.store.db.write_transaction():
      3      ...
      4  with spack.store.db.read_transaction():
         ...
      ```
      
      The WriteTransaction on line 2 had no way of knowing that its
      `__exit__()` call was the last *write* in the nesting, and it would skip
      calling its write function.
      
      The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know
      how to write, and the file would never be written.
      
      The DB would be correct in memory, but the `ReadTransaction` on line 4
      would re-read the whole DB assuming that other processes may have
      modified it.  Since the DB was never written, we got stale data.
      
      - [x] Make `Lock.release_write()` return `True` whenever we release the
            *last write* in a nest.
      bb517fdb
    • Todd Gamblin's avatar
      lock transactions: fix non-transactional writes · eb8fc4f3
      Todd Gamblin authored
      Lock transactions were actually writing *after* the lock was
      released. The code was looking at the result of `release_write()` before
      writing, then writing based on whether the lock was released.  This is
      pretty obviously wrong.
      
      - [x] Refactor `Lock` so that a release function can be passed to the
            `Lock` and called *only* when a lock is really released.
      
      - [x] Refactor `LockTransaction` classes to use the release function
        instead of checking the return value of `release_read()` / `release_write()`
      eb8fc4f3
Loading