Switch ChronologyProtector from redis to memcached
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aaron
	Aug 3 2022, 5:46 AM

Description

This will save a ~0.5ms Redis connection on most requests (those that write to FileRepo will still use the redis lock manager).

Details

	Project	Branch	Lines +/-	Subject
	operations/mediawiki-config	master	+1 -1	Switch $wgChronologyProtectorStash to "mcrouter"

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T302623 FY2022-2023: Improve Backend Pageview Timing
		Resolved		Krinkle	T314453 Switch ChronologyProtector from redis to memcached

Event Timeline

aaron created this task.Aug 3 2022, 5:46 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 3 2022, 5:46 AM

Afaik Redis lock manager uses the rdb* hosts instead (part of dedicated "redis_misc" cluster; as opposed to redis_sessions which is colocated on memc hosts).

See also:

I believe (but let's emperically confrim after this is done) that if we move ChronologyProtector to memc, then redis_sessions can be decom'ed as-is and thus remove the need to for the migration at T280586: Move "redis_sessions" to "redis_misc" cluster and also allows removal of nutcracker as afaik rdb/lockmanager doesn't use that.

Krinkle added a project: serviceops.Aug 3 2022, 5:52 AM

Krinkle added a parent task: T302623: FY2022-2023: Improve Backend Pageview Timing.

Krinkle updated the task description. (Show Details)

• Di3sel1975 added a subtask: T314452: VisualEditor returns error in blkwiki.Aug 3 2022, 6:34 AM

Aklapper removed a subtask: T314452: VisualEditor returns error in blkwiki.Aug 3 2022, 6:41 AM

Krinkle moved this task from Inbox, needs triage to To-do: Goals, prioritized next 4 Quarters on the Performance-Team board.Aug 8 2022, 7:15 PM

Change 824556 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[operations/mediawiki-config@master] Switch $wgChronologyProtectorStash to "mcrouter"

https://gerrit.wikimedia.org/r/824556

gerritbot added a project: Patch-For-Review.Aug 18 2022, 7:44 PM

Logstash mwdebug dashboard, query (channel:DBReplication AND ChronologyProtector) OR channel:redis OR channel:memcached

Before

  [memcached] MainWANObjectCache using store MemcachedPeclBagOStuff
  [memcached] MemcachedPeclBagOStuff::initializeClient: initializing new client instance.
  [memcached] MemcachedPeclBagOStuff debug: getMulti(WANCache:global:rdbms-server-states:1:db1157:0-1-2-3-4-5|#|v)
- [DBReplication] ChronologyProtector using store RedisBagOStuff
- [redis] RedisBagOStuff debug: get(global:Wikimedia\Rdbms\ChronologyProtector:…:v2) on /var/run/nutcracker/redis_eqiad.sock: success
  [DBReplication] Wikimedia\Rdbms\ChronologyProtector::applySessionReplicationPosition: DEFAULT (db1157) has no position

After

  [memcached] MainWANObjectCache using store MemcachedPeclBagOStuff
  [memcached] MemcachedPeclBagOStuff::initializeClient: initializing new client instance.
  [memcached] MemcachedPeclBagOStuff debug: getMulti(WANCache:global:rdbms-server-states:1:db1157:0-1-2-3-4-5|#|v)
+ [DBReplication] ChronologyProtector using store MemcachedPeclBagOStuff
+ [memcached] MemcachedPeclBagOStuff debug: get(global:Wikimedia\Rdbms\ChronologyProtector:…:v2)
+ [memcached] MemcachedPeclBagOStuff debug: result: NOT FOUND
  [DBReplication]  Wikimedia\Rdbms\ChronologyProtector::applySessionReplicationPosition: DEFAULT (db1157) has no position

In T280586#8168908 for T280586: Move "redis_sessions" to "redis_misc" cluster, @Krinkle wrote:

[…] the last consumer is gone

Details and references to the various tasks that made this possible can be found at: https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_sessions