Page MenuHomePhabricator

[CLIENT][SW] Confirm Wikibase LuaFunctionCallTracker to statsd naming convention
Open, Needs TriagePublic

Description

The stats from LuaFunctionCallTracker are perhaps a bit too deep and integrated into the global namespace of stats messages. For example:

enwiki.wikibase.client.scribunto.wikibase.getEntityStatements.cache_miss:100|c
enwiki.wikibase.client.scribunto.wikibase.getEntityStatements.call:100|c

wikipedia.wikibase.client.scribunto.wikibase.getEntity.call:100|c
wikipedia.wikibase.client.scribunto.wikibase.getEntityStatements.call:100|c

Compared to:

resourceloader_build.all:0.12707710266113|ms
resourceloader_build.user_options:0.12707710266113|ms

wanobjectcache.resourceloader_titleinfo.hit.volatile:0.41294097900391|ms
wanobjectcache.filerepo_file.hit.good:0.50878524780273|ms

In particular:

  • The fragments (sitegroup, wikiid) are top-level metrics whereas these should go to the end to avoid conflicts with other metrics.
  • There is no containing metric for LuaFunctionCallTracker which makes these difficult to discover and organize.

Source code:

I suggest the following metric pattern instead:

wikibase_client_lua_tracker_by_wiki . <key> . <wiki>
wikibase_client_lua_tracker_by_sitegroup . <key> . <sitegroup>

Where key would be a single segment, eg. dots converted to underscores using strtr( $key, '.', '_' ) . The Grafana dashboard can continue to query and plot multiple of these using wildcards, which are supported on segments as well. We can even copy over the existing data if preferred, using ad-hoc scripting on the Graphite which I can help with (and we've done this few times before).

This is also in preparation for Prometheus, to make the metric ready for the conversion. From a 5 minute capture in production, this was the only metric source I found that was not grouping its metrics or that was using many more subsegments than seem to logically expected from the underlyng source code.

Event Timeline

Is this “statsd naming convention” documented anywhere? To me this task seems counterintuitive.

It's not documented to my knowledge, but it is a pattern that is followed quite consistently throughout our software and elsewhere in the industry as relating to how StatsD, Graphite and Prometheus are used.

Metrics follow a left-to-right hierarchy. In Prometheus this is done with underscores, in StatsD this is done with dots. There is generally one main segment at the start for the overal service or component, then a segment for the metric being measured, and then labels (Prometheus), hash tags (dogstatsd), or subcomponents (statsd/Graphite).

The labels/subcomponents should only be used to represent subslices of the same metric, not to create entirely different metrics. So, for example, you would have foo_bar_request_count.wiki and foo_bar_request_size.wiki and not foo_bar_request.wiki.request_count and `foo_bar_request.wiki .request_size. This is also reflected on https://prometheus.io/docs/practices/naming/, because incompatible data structures would otherwise clash and result in non-sensical visualisations when later segments are left out of the query with a wildcard. More on Graphite/Statsd: https://matt.aimonetti.net/posts/2013-06-practical-guide-to-graphite-monitoring/, though practically any software with a statsd exporter will serve as equally good example to find these conventions applied.

This ensures maxium benefit of the wider ecosystem through Grafana, such as autocompletion, wildcard queries, and templates. And also reduces chances of conflicts with unrelated metrics clashing in the same timeseries database directory. Remember that for Graphite, each dot-component is a subdirectory on disk where a file is stored, so enwiki.wikibase.client.scribunto.wikibase.getEntityStatements.cache_miss gets stored in /var/lib/carbon/enwiki/wikibase/client/scribunto/wikibase/getEntityStatements/cache_miss/rate.wsp, for example.

ArthurTaylor renamed this task from Confirm Wikibase LuaFunctionCallTracker to statsd naming convention to [CLIENT][SW] Confirm Wikibase LuaFunctionCallTracker to statsd naming convention.Apr 2 2024, 3:02 PM
ArthurTaylor moved this task from Incoming to [DOT] By Project on the wmde-wikidata-tech board.