In tasks like T401705 we're getting good observability for xLab events flowing through our systems. But we'd also like some more general baselines. Specifically, in this task we'd like to track:
How many user agents load the xLab JS SDK?
Open Question: do we want to know this only for agents that are in-sample? Or does WMF more broadly care about this kind of metric and we should set up something more generic?
Two signals that could help here are:
- send an image-based update to statsd when HTML renders, so even non-JS clients are counted (look into how ad-block interacts with this)
- increment a counter as soon as mw.loader.using( 'ext.xLab' ...
TODO: check with other folks interested in these kinds of baselines about what might be useful broadly for WMF to know.
Background
Timo was telling us of a mechanism he used to figure out how many users were using Grade C browsers vs. Grade A browsers some years ago. A static inline HTML <script> tag would emit a counter to statsv like "Page loaded" and then later in JS another counter would go out like "And now JS has loaded". Analyzing the difference would show how many user agents either leave before JS loads or can not run JS at all.
@phuedx was thinking this mechanism would be very useful for us to baseline how many user agents can load xLab client code. Similarly we'd have an inline <script> tag emit some event like "Page loaded" (this could even check whether Varnish has put it in an experiment bucket). And then when xLab clients are called maybe we'd emit another event. We could run this and other baselines once a year and keep the data somewhere to serve as intuition for people designing and running experiments.