In https://gerrit.wikimedia.org/r/c/mediawiki/core/+/452278 I mistakenly assumed that 'setup' would get called for each iteration. My goal there was to randomize the input in each loop so I could avoid any caches.
|mediawiki/core||master||+14 -2||benchmarker: Implement setupEach for per-iteration setup|
Most benchmarkers I've used do not have this option. It might be worth considering whether the iterated code can be changed in a way to produce the expected results "within" the current model. I don't have many ideas off-hand, but one thing might be to have a way to disable the (static?) caches. Would that suffice?
Either way, I'd support addition of an option to perform setup per iteration. Perhaps a callable option, setupEach?