feat: use spack ci generate to setup build pipeline

Rather than having a single long build, we can split it up in different jobs with spack ci generate. This will be more resilient against failures. We can also run the builds inside a regular job instead of inside the container, and then only pull from the build cache once we build the container.

.gitlab-ci.yml

@@ -230,6 +230,7 @@ generate-pipeline:
     name: ${CI_REGISTRY_IMAGE}/${BASE_IMAGE}:${INTERNAL_TAG}
     entrypoint: [""]
   before_script:
+    - apt-get install -yqq python3-boto3
     - git clone https://github.com/${SPACK_ORGREPO}.git spack
     - git -C spack checkout ${SPACK_VERSION}
     - . "./spack/share/spack/setup-env.sh"