ID
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aaron
	Oct 16 2019, 7:54 PM

Description

This would give insight into basic questions like "write rate to redis in ops/sec or bytes/sec" and "what are the main use cases". The lack of information is an obstacle to alternative store and multi-DC discussions (e.g. main stash migration work).

We'd have to exclude or have logic to handle WAN cache key prefixes.

Outcome

Have gauge metrics in Grafana for bytes written from Memcached via BagOStuff, by keygroup level (including WANObjectCache).
Have gauge metrics in Grafana for bytes read from Memcached via BagOStuff, by keygroup level (including WANObjectCache).).

See T254608 for the WANObjectCache-specific version of this (might be inspiration and/or become obsolete).

Details

	Subject	Repo	Branch	Lines +/-
	objectcache: make BagOStuff key encoding more consistent	mediawiki/core	master	+756 -285
	objectcache: add statsd key metrics to BagOStuff classes	mediawiki/core	master	+392 -95

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		aaron	T235705 Add BagOStuff metrics for read/write operations (bytes, key "class") grouped by type/ID
		Resolved		aaron	T254608 Monitor bytes written and read via WANObjectCache by key group

Event Timeline

aaron created this task.Oct 16 2019, 7:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 16 2019, 7:54 PM

Krinkle moved this task from General to libs/objectcache on the MediaWiki-libs-BagOStuff board.Oct 16 2019, 9:07 PM

aaron updated the task description. (Show Details)Oct 23 2019, 6:02 PM

aaron added a project: Platform Engineering.Oct 23 2019, 6:16 PM

Untagging Platform Engineering for now, but please feel free to retag when there is something specific for us to do.

• Gilles moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.Oct 23 2019, 7:55 PM

aaron added a subscriber: Eevans.Oct 24 2019, 7:44 AM

aaron mentioned this in T236414: CPT review/work for MediaWiki caching class maintenance ramp-up.Oct 24 2019, 5:20 PM

Change 589465 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: make BagOStuff key encoding more consistent

https://gerrit.wikimedia.org/r/589465

gerritbot added a project: Patch-For-Review.Apr 17 2020, 5:38 AM

Krinkle triaged this task as Low priority.May 13 2020, 11:42 PM

Krinkle moved this task from Doing (old) to Blocked (old) on the Performance-Team board.

Is this a blocker for migrating redis/mainstash to something else? Do we already know that the overall redis read traffic (which we can measure outside MW) would be too much for e.g. db-replicated (or someting else) to handle?

Krinkle added a parent task: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.May 13 2020, 11:51 PM

Change 589465 merged by jenkins-bot:
[mediawiki/core@master] objectcache: make BagOStuff key encoding more consistent

https://gerrit.wikimedia.org/r/589465

Maintenance_bot removed a project: Patch-For-Review.Dec 11 2020, 9:10 PM

Krinkle removed a parent task: T212129: Move MainStash out of Redis to a simpler multi-dc aware solution.Dec 22 2020, 8:45 PM

Change 652572 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] objectcache: add statsd key metrics to BagOStuff classes

https://gerrit.wikimedia.org/r/652572

gerritbot added a project: Patch-For-Review.Dec 30 2020, 5:33 PM

Krinkle updated the task description. (Show Details)Jan 19 2021, 12:16 AM

Krinkle added a subtask: T254608: Monitor bytes written and read via WANObjectCache by key group.

Change 652572 merged by jenkins-bot:
[mediawiki/core@master] objectcache: add statsd key metrics to BagOStuff classes

https://gerrit.wikimedia.org/r/652572

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.28; 2021-01-26).Jan 26 2021, 1:00 AM

Maintenance_bot removed a project: Patch-For-Review.Jan 26 2021, 1:11 AM

aaron updated the task description. (Show Details)Jan 26 2021, 4:11 AM

WMDE-Fisch mentioned this in T273061: Error running update.php in Modifying exptime field of table objectcache.Jan 27 2021, 1:53 PM

It appears rMW57325ba3bdce: objectcache: add statsd key metrics to BagOStuff classes is causing notices about undefined indexes:

Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1083
(10.0.2.15)
Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1098
(10.0.2.15)

I can only reproduce this on MediaWiki-Vagrant, which comes with a Redis setup out of the box. I don't know if this means Vagrant is misconfigured, or the aforementioned change is incomplete.

In T235705#6787784, @Mainframe98 wrote:
It appears rMW57325ba3bdce: objectcache: add statsd key metrics to BagOStuff classes is causing notices about undefined indexes:
Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1083
(10.0.2.15)
Notice: Undefined index: wiki:user-quicktouched:id:1 in /vagrant/mediawiki/includes/libs/objectcache/MediumSpecificBagOStuff.php on line 1098
(10.0.2.15)
I can only reproduce this on MediaWiki-Vagrant, which comes with a Redis setup out of the box. I don't know if this means Vagrant is misconfigured, or the aforementioned change is incomplete.

See https://gerrit.wikimedia.org/r/c/mediawiki/core/+/660054

Mainframe98 mentioned this in T273634: MediumSpecificBagOStuff undefined index errors on article pages.Feb 2 2021, 4:28 PM

KartikMistry mentioned this in T273364: In Main page of Section translation test server, i'm having a "Undefined index" error.Feb 4 2021, 1:16 PM

aaron closed subtask T254608: Monitor bytes written and read via WANObjectCache by key group as Resolved.Feb 22 2021, 7:14 PM

aaron closed this task as Resolved.Feb 22 2021, 7:23 PM

aaron updated the task description. (Show Details)