Background benchmark job leads to 12GB download and fills up persistent cache on runner machine
We are using the full 1TB /scratch allocation on the eic-n0 runner machine. Culprit is the background benchmark which each time downloads 12GB of data which fills up /scratch quicker than the cleanup cron-job clears old files. This is a critical issue as it prevents all benchmarks from running!
This is also an issue besides the disk storage limit, as it involves a lot of unnecessary bandwidth to simulate a relatively smaller number of events. This should be replaced by a direct read from S3 (rather than a cleanup within the job as we should try to avoid using unnecessary bandwidth).
I will disable the benchmark for now and file a new issue to change the benchmark into a direct read from S3.
Edited by Sylvester Joosten