physics_benchmarks issueshttps://eicweb.phy.anl.gov/EIC/benchmarks/physics_benchmarks/-/issues2021-10-02T00:15:49Zhttps://eicweb.phy.anl.gov/EIC/benchmarks/physics_benchmarks/-/issues/23Use mcparticles instead of mcparticles22021-10-02T00:15:49ZSylvester JoostenUse mcparticles instead of mcparticles2No need for the MCCopier anymore with recent versions of Juggler, better to just propagate the input structure.No need for the MCCopier anymore with recent versions of Juggler, better to just propagate the input structure.Sylvester JoostenSylvester Joostenhttps://eicweb.phy.anl.gov/EIC/benchmarks/physics_benchmarks/-/issues/8Benchmark definition standard2020-12-23T23:36:39ZWhitney ArmstrongBenchmark definition standardHow can we define each benchmark and the metric on which it succeeds?
For example, detection efficiency might detect 80% of events with some Q2 cut and we want it to fail lower than 95%. Could we just have a json file like the following...How can we define each benchmark and the metric on which it succeeds?
For example, detection efficiency might detect 80% of events with some Q2 cut and we want it to fail lower than 95%. Could we just have a json file like the following?
```
{ "name": "My Q2 cut",
"description":"Some Q2 cut that we expect high eff.",
"quantity":"efficiency",
"benchmark":"0.95",
"value":"0.80"
}
```
Should we think of this as a "benchmark" or a "test"?
I guess a "benchmark" could be comprised of one or more of these "tests"
```
{ benchmark : "DVCS in central",
test_results: [
{ "name": "My Q2 cut",
"description":"Some Q2 cut that we expect high eff.",
"quantity":"efficiency",
"goal_threshold":"0.95",
"value":"0.80",
"weight": "1.0"
},
{ "name": "Coplanarity analysis",
...
},
...
],
performance_limit "4.5"
performance_goal : "4",
performance: "4.1",
successful_goals: "5",
total_goals: "6"
}
```
where `performance_limit` is computed from the weights:
```math
P_{limit} = \sum_{tests}^i w_i
```
and the actual performance includes only passing tests:
```math
P = \sum_{tests passed}^i w_i\
```
This assumes a all tests are pass/fail can probably be relaxed to a measure between [0,1].
Thoughts? @sly2j @cpeng @jihee.kim @PolakovicWhitney ArmstrongWhitney Armstrong2020-12-01