Page MenuHomePhabricator

StatsD metrics for WANObjectCache misreported for key components containing a dot
Closed, ResolvedPublic

Description

The stats for RevisionStore::getKnownCurrentRevision use as first makeKey component the string revision-row-1.29.

This works fine for BagOStuff backends. Any theoretical issue might might exist with a backend using - or . for special purposes would have a makeKeyInternal method that escapes or substitutes these as needed.

However, when writing statistics to StatsD, the dot definitely has a special meaning, it creates a metric namespace. For Graphite, that means an extra directory on disk, and thus when querying for metrics in the same namespace, they are not picked up due to not being in the expected directory.

Example:

		$key = $this->makeGlobalKey( 'revision-row-1.29', 'foo', 'bar' ); # "revision-row-1.29:foo:bar"
		$kClass = $this->determineKeyClassForStats( $key ); # => explode( ':', $key )[0] => "revision-row-1.29"
		$this->stats->increment( "wanobjectcache.$kClass.somemetric" );

This results in wanobjectcache . revision_row_1 . 29 . somemetric, which does not get returned by Graphite when searching for wanobjectcache . * . somemetric.

Sub tasks:

  • Fix the bug with regression test.
  • Prune or migrate the inaccessible data on graphite1003/graphite2004:/var/lib/carbon/whisper/MediaWiki/wanobjectcache/revision_row_1/29. Ideally in the minutes leading up to rolling out the patch.
  • Once rolled out, verify that the wanobjectcache Grafana dashboard provides statistics for revision-row keys from RevisionStore::getKnownCurrentRevision.

Event Timeline

Krinkle created this task.Sep 14 2019, 1:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 14 2019, 1:57 AM
Krinkle claimed this task.Sep 14 2019, 1:57 AM
Krinkle added a project: Performance-Team.
Krinkle moved this task from Untriaged to libs/objectcache on the MediaWiki-Cache board.

Change 536724 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] objectcache: Escape dots from cache keys in StatsD metrics

https://gerrit.wikimedia.org/r/536724

Krinkle updated the task description. (Show Details)Sep 14 2019, 2:07 AM
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
Krinkle triaged this task as Normal priority.Sep 14 2019, 6:15 PM

Change 536724 merged by jenkins-bot:
[mediawiki/core@master] objectcache: Escape dots from cache keys in StatsD metrics

https://gerrit.wikimedia.org/r/536724

Change 537765 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@wmf/1.34.0-wmf.23] objectcache: Escape dots from cache keys in StatsD metrics

https://gerrit.wikimedia.org/r/537765

Change 537765 merged by jenkins-bot:
[mediawiki/core@wmf/1.34.0-wmf.23] objectcache: Escape dots from cache keys in StatsD metrics

https://gerrit.wikimedia.org/r/537765

Krinkle closed this task as Resolved.Sep 19 2019, 5:26 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-19T17:27:00Z] <krinkle@deploy1001> Synchronized php-1.34.0-wmf.23/includes/libs/objectcache/wancache: 2e910c9d3f8c04f7db, T232907 (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2019-09-19T17:43:06Z] <Krinkle> Move whisper/MediaWiki/wanobjectcache/revision_row_1/29 to whisper/MediaWiki/wanobjectcache/revision_row_1_29 on graphite1004 and graphite2003 (T232907)