Resolve "Background benchmark job leads to 12GB download and fills up persistent cache on runner machine"

For this to work, it needs some custom code to create ad-hoc branches with merged PR state. Let's disable this for now.