I initially thought it was just a change of color due to the order of the metrics being indeterministic. But, that's not it.

The closing metric visible in figure A is missing from figure B. The logging metric visible in figure B is missing from figure A.

It seems something inside Prometheus, or the collectors, is causing one of the two to constantly be deleted or absent, not even showing in historical data, and then a minute later, its history and current value are back, with another metric's current/historic values missing instead.

The prometheus.svc endpoint in eqiad and codfw is backed by two independent Prometheus servers scraping the same targets. What I suspect has happened is that one of the two servers "catched" workers in state closing or logging while the other didn't. This also suggests to me the exporter doesn't report all metrics it knows about all the time, which leads me to believe that mod_status believes that way (i.e. when no workers are in state closing they are not reported at all).

One potential fix would be for the exporter to report all metrics it knows about all the time, if my assumptions are correct.