Page MenuHomePhabricator

Phasing out "redis_sessions" MediaWiki cluster
Open, MediumPublic

Description

This task is about phasing out the co-located Redis instances on MW's memcached servers, which was historically used for MW's session storage and MW's main stash, both of which have or are being migrated to other backends.

Background

We have been discussing this topic for a few quarters now, it is time we do the final pushes to get to it. Please attach any relevant tasks and update the description with a checklist accordingly.

We are planning to keep this cluster to version Redis 2.8, even after we upgrade those hosts to buster, but this is putting an administrative overhead of keeping and maintaining a very old Redis version T265643

Related Objects

StatusSubtypeAssignedTask
OpenNone
Openaaron
ResolvedEevans
Resolvedaaron
OpenNone
ResolvedPapaul
OpenNone
DeclinedNone
ResolvedMarostegui
ResolvedJclark-ctr
ResolvedMarostegui
ResolvedMarostegui
ResolvedRequestwiki_willy
OpenMarostegui
OpenTrizek-WMF
ResolvedCmjohnson
OpenRobH
ResolvedMarostegui
StalledMarostegui
OpenCmjohnson
Resolveddcaro
OpenBPirkle
OpenMarostegui
OpenNone
Resolvedjijiki
Resolvedjijiki

Event Timeline

jijiki created this task.Nov 9 2020, 5:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 9 2020, 5:14 PM
Gilles moved this task from Inbox to Radar on the Performance-Team board.Nov 9 2020, 7:40 PM
Gilles edited projects, added Performance-Team (Radar); removed Performance-Team.
Krinkle renamed this task from Phasing out MediaWiki Redis to Phasing out "redis_sessions" MediaWiki cluster.Nov 9 2020, 7:40 PM
Krinkle updated the task description. (Show Details)
Krinkle added a subscriber: Krinkle.Nov 9 2020, 7:43 PM

Clarified title and summary based on chat with Effie on IRC.

The two remaining uses for a simpler non-replicated are ChronologyProtector and (probably) CentralAuth tokens. I suppose these could be folded in to dedicated Redis cluster we have for MediaWiki, on the the "rdb*" hosts. This is used for MW's lock manager, and was (previously) used for JobQueue. The CP and CA uses could be migrated there fairly easily I think.

Migrating CP or CA to something else is also an option long-term, but not a priority I think as we have plenty of other more impactful things to work on, and this is mostly working fine as it is, and still a win-win in terms of opeational complexity and maintenance cost.

jijiki added a comment.EditedNov 9 2020, 9:15 PM

Clarified title and summary based on chat with Effie on IRC.

The two remaining uses for a simpler non-replicated are ChronologyProtector and (probably) CentralAuth tokens. I suppose these could be folded in to dedicated Redis cluster we have for MediaWiki, on the the "rdb*" hosts. This is used for MW's lock manager, and was (previously) used for JobQueue. The CP and CA uses could be migrated there fairly easily I think.

Just a correction, so not to confuse the various redis clusters 🙂 . The rdb* hosts constitute what we call the misc redis cluster, currently used by changeprop, ORES and by our docker-registry. MW's lock manager consists of 3 redis hosts from the 'session redis' cluster.

Please note that this cluster is not highly available, so if one master goes down, it will have to be manually substituted by the secondary one. Moreover, data in this cluster is dc-local and not replicated.

jijiki triaged this task as Medium priority.Nov 10 2020, 3:59 PM

Change 640576 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/mediawiki-config@master] ProductionServices: Document hostname of redis_lock hosts

https://gerrit.wikimedia.org/r/640576

Change 640576 merged by jenkins-bot:
[operations/mediawiki-config@master] ProductionServices: Document hostname of redis_lock hosts

https://gerrit.wikimedia.org/r/640576

Change 646638 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] redis: define redis version on buster

https://gerrit.wikimedia.org/r/646638

Change 647197 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] redis: define redis version on buster for multidc

https://gerrit.wikimedia.org/r/647197

Change 646638 abandoned by Effie Mouzeli:
[operations/puppet@production] redis: define redis version on buster

Reason:
abandoned for 647197

https://gerrit.wikimedia.org/r/646638

Change 647197 merged by Effie Mouzeli:
[operations/puppet@production] redis: define redis version on buster for multidc

https://gerrit.wikimedia.org/r/647197