Page MenuHomePhabricator

Add BagOStuff metrics for read/write operations (bytes, key "class") grouped by type/ID
Closed, ResolvedPublic

Description

This would give insight into basic questions like "write rate to redis in ops/sec or bytes/sec" and "what are the main use cases". The lack of information is an obstacle to alternative store and multi-DC discussions (e.g. main stash migration work).

We'd have to exclude or have logic to handle WAN cache key prefixes.

Outcome
  • Have gauge metrics in Grafana for bytes written from Memcached via BagOStuff, by keygroup level (including WANObjectCache).
  • Have gauge metrics in Grafana for bytes read from Memcached via BagOStuff, by keygroup level (including WANObjectCache).).

See T254608 for the WANObjectCache-specific version of this (might be inspiration and/or become obsolete).

Event Timeline

CCicalese_WMF added a subscriber: CCicalese_WMF.

Untagging Platform Engineering for now, but please feel free to retag when there is something specific for us to do.

Change 589465 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: make BagOStuff key encoding more consistent

https://gerrit.wikimedia.org/r/589465

Krinkle moved this task from Doing (old) to Blocked (old) on the Performance-Team board.
Krinkle raised the priority of this task from Low to High.May 13 2020, 11:50 PM
Krinkle moved this task from Blocked (old) to Doing (old) on the Performance-Team board.
Krinkle added a subscriber: Krinkle.

Is this a blocker for migrating redis/mainstash to something else? Do we already know that the overall redis read traffic (which we can measure outside MW) would be too much for e.g. db-replicated (or someting else) to handle?

Change 589465 merged by jenkins-bot:
[mediawiki/core@master] objectcache: make BagOStuff key encoding more consistent

https://gerrit.wikimedia.org/r/589465

Change 652572 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: add statsd key metrics to BagOStuff classes

https://gerrit.wikimedia.org/r/652572

Change 652572 merged by jenkins-bot:
[mediawiki/core@master] objectcache: add statsd key metrics to BagOStuff classes

https://gerrit.wikimedia.org/r/652572

It appears rMW57325ba3bdce: objectcache: add statsd key metrics to BagOStuff classes is causing notices about undefined indexes:

Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1083
(10.0.2.15)
Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1098
(10.0.2.15)

I can only reproduce this on MediaWiki-Vagrant, which comes with a Redis setup out of the box. I don't know if this means Vagrant is misconfigured, or the aforementioned change is incomplete.

It appears rMW57325ba3bdce: objectcache: add statsd key metrics to BagOStuff classes is causing notices about undefined indexes:

Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1083
(10.0.2.15)
Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1098
(10.0.2.15)

I can only reproduce this on MediaWiki-Vagrant, which comes with a Redis setup out of the box. I don't know if this means Vagrant is misconfigured, or the aforementioned change is incomplete.

See https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660054

aaron updated the task description. (Show Details)