Skip to content
Snippets Groups Projects

feat: let gitlab parallel drive multi-node snakemake

Open feat: let gitlab parallel drive multi-node snakemake
snakemake-batch into master
2 unresolved threads
Open Wouter Deconinck requested to merge snakemake-batch into master
2 unresolved threads

We are using single job slots to run multi-core jobs, which is oversubscribing modes. This MR aims to let gitlab know how many nodes we are using for parallel (or not) snakemake batches without oversubscribing.

Merge request reports

Approval is optional
Merge blocked: 2 checks failed
Unresolved discussions must be resolved.
Merge request must be rebased, because a fast-forward merge is not possible.

Merge details

  • The source branch is 66 commits behind the target branch.
  • 1 commit will be added to master.
  • Source branch will be deleted.

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
benchmarks/ecal_gaps/config.yml
11 11 stage: benchmarks
12 12 needs:
13 13 - ["sim:ecal_gaps"]
14 parallel: 8
  • Dmitry Kalinkin
    Dmitry Kalinkin @veprbl started a thread on the diff
    • benchmarks/ecal_gaps/config.yml
    11 11 stage: benchmarks
    12 12 needs:
    13 13 - ["sim:ecal_gaps"]
    14 parallel: 8
    14 15 script:
    15 16 - ln -s $LOCAL_DATA_PATH/input input
    16 17 - export PYTHONUSERBASE=$LOCAL_DATA_PATH/deps
    17 18 - pip install -r benchmarks/ecal_gaps/requirements.txt
    18 - snakemake $SNAKEMAKE_FLAGS --cores 8 results/epic_inner_detector/ecal_gaps
    19 - snakemake $SNAKEMAKE_FLAGS --cores 1 --batch results/epic_inner_detector/ecal_gaps=$CI_NODE_INDEX/$CI_NODE_TOTAL
    • Dmitry Kalinkin
      Dmitry Kalinkin @veprbl
      Maintainer
      Suggested change
      19 - snakemake $SNAKEMAKE_FLAGS --cores 1 --batch results/epic_inner_detector/ecal_gaps=$CI_NODE_INDEX/$CI_NODE_TOTAL
      19 - snakemake $SNAKEMAKE_FLAGS --cores 5 --batch results/epic_inner_detector/ecal_gaps=$CI_NODE_INDEX/$CI_NODE_TOTAL
    • Please register or sign in to reply
    Please register or sign in to reply