-
- Downloads
fetching: S3 upload and download (#11117)
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket. High level to-do list: - [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs. - [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs. - [x] Implement S3 source fetching - [x] Implement S3 binary mirror indexing - [x] Implement S3 binary package fetching - [x] Implement S3 source pushing - [x] Implement S3 binary package pushing Important details: * refactor URL handling to handle S3 URLs and mirror URLs more gracefully. - updated parse() to accept already-parsed URL objects. an equivalent object is returned with any extra s3-related attributes intact. Objects created with urllib can also be passed, and the additional s3 handling logic will still be applied. * update mirror schema/parsing (mirror can have separate fetch/push URLs) * implement s3_fetch_strategy/several utility changes * provide more feature-complete S3 fetching * update buildcache create command to support S3 * Move the core logic for reading data from S3 out of the s3 fetch strategy and into the s3 URL handler. The s3 fetch strategy now calls into `read_from_url()` Since read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant. It's not clear whether the ideal design is to have S3 fetching functionality in a fetch strategy, directly implemented in read_from_url, or both. * expanded what can be passed to `spack buildcache` via the -d flag: In addition to a directory on the local filesystem, the name of a configured mirror can be passed, or a push URL can be passed directly.
Showing
- .gitlab-ci.yml 5 additions, 0 deletions.gitlab-ci.yml
- bin/rebuild-index.sh 1 addition, 1 deletionbin/rebuild-index.sh
- lib/spack/spack/binary_distribution.py 149 additions, 84 deletionslib/spack/spack/binary_distribution.py
- lib/spack/spack/caches.py 11 additions, 5 deletionslib/spack/spack/caches.py
- lib/spack/spack/cmd/buildcache.py 28 additions, 0 deletionslib/spack/spack/cmd/buildcache.py
- lib/spack/spack/cmd/checksum.py 2 additions, 2 deletionslib/spack/spack/cmd/checksum.py
- lib/spack/spack/cmd/create.py 2 additions, 1 deletionlib/spack/spack/cmd/create.py
- lib/spack/spack/cmd/mirror.py 99 additions, 26 deletionslib/spack/spack/cmd/mirror.py
- lib/spack/spack/cmd/url.py 3 additions, 5 deletionslib/spack/spack/cmd/url.py
- lib/spack/spack/fetch_strategy.py 97 additions, 7 deletionslib/spack/spack/fetch_strategy.py
- lib/spack/spack/mirror.py 221 additions, 9 deletionslib/spack/spack/mirror.py
- lib/spack/spack/s3_handler.py 92 additions, 0 deletionslib/spack/spack/s3_handler.py
- lib/spack/spack/schema/mirrors.py 13 additions, 1 deletionlib/spack/spack/schema/mirrors.py
- lib/spack/spack/stage.py 103 additions, 15 deletionslib/spack/spack/stage.py
- lib/spack/spack/test/cmd/pkg.py 2 additions, 0 deletionslib/spack/spack/test/cmd/pkg.py
- lib/spack/spack/test/config.py 1 addition, 0 deletionslib/spack/spack/test/config.py
- lib/spack/spack/test/llnl/util/lock.py 2 additions, 0 deletionslib/spack/spack/test/llnl/util/lock.py
- lib/spack/spack/test/stage.py 2 additions, 0 deletionslib/spack/spack/test/stage.py
- lib/spack/spack/util/s3.py 44 additions, 0 deletionslib/spack/spack/util/s3.py
- lib/spack/spack/util/url.py 175 additions, 0 deletionslib/spack/spack/util/url.py
Loading
Please register or sign in to comment