We are only running benchmarks on default builds, not nightly. This means that lots can go wrong when we update the EICRECON_VERSION since it could cover a large number of commits. Surfacing errors earlier with benchmarks against nightly is preferred (even though we should really trigger the benchmarks from EICrecon).
