Skip to content
Snippets Groups Projects
  • Omar Padron's avatar
    fd58c98b
    fetching: S3 upload and download (#11117) · fd58c98b
    Omar Padron authored
    This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket.
    
    High level to-do list:
    
    - [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs.
    - [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs.
    - [x] Implement S3 source fetching
    - [x] Implement S3 binary mirror indexing
    - [x] Implement S3 binary package fetching
    - [x] Implement S3 source pushing
    - [x] Implement S3 binary package pushing
    
    Important details:
    
    * refactor URL handling to handle S3 URLs and mirror URLs more gracefully.
      - updated parse() to accept already-parsed URL objects.  an equivalent object
        is returned with any extra s3-related attributes intact.  Objects created with
        urllib can also be passed, and the additional s3 handling logic will still be applied.
    
    * update mirror schema/parsing (mirror can have separate fetch/push URLs)
    * implement s3_fetch_strategy/several utility changes
    * provide more feature-complete S3 fetching
    * update buildcache create command to support S3
    
    * Move the core logic for reading data from S3 out of the s3 fetch strategy and into
      the s3 URL handler.  The s3 fetch strategy now calls into `read_from_url()` Since
      read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant.  It's
      not clear whether the ideal design is to have S3 fetching functionality in a fetch
      strategy, directly implemented in read_from_url, or both.
    
    * expanded what can be passed to `spack buildcache` via the -d flag: In addition
      to a directory on the local filesystem, the name of a configured mirror can be
      passed, or a push URL can be passed directly.
    fd58c98b
    History
    fetching: S3 upload and download (#11117)
    Omar Padron authored
    This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket.
    
    High level to-do list:
    
    - [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs.
    - [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs.
    - [x] Implement S3 source fetching
    - [x] Implement S3 binary mirror indexing
    - [x] Implement S3 binary package fetching
    - [x] Implement S3 source pushing
    - [x] Implement S3 binary package pushing
    
    Important details:
    
    * refactor URL handling to handle S3 URLs and mirror URLs more gracefully.
      - updated parse() to accept already-parsed URL objects.  an equivalent object
        is returned with any extra s3-related attributes intact.  Objects created with
        urllib can also be passed, and the additional s3 handling logic will still be applied.
    
    * update mirror schema/parsing (mirror can have separate fetch/push URLs)
    * implement s3_fetch_strategy/several utility changes
    * provide more feature-complete S3 fetching
    * update buildcache create command to support S3
    
    * Move the core logic for reading data from S3 out of the s3 fetch strategy and into
      the s3 URL handler.  The s3 fetch strategy now calls into `read_from_url()` Since
      read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant.  It's
      not clear whether the ideal design is to have S3 fetching functionality in a fetch
      strategy, directly implemented in read_from_url, or both.
    
    * expanded what can be passed to `spack buildcache` via the -d flag: In addition
      to a directory on the local filesystem, the name of a configured mirror can be
      passed, or a push URL can be passed directly.