🤖 Deploy new redis cluster for main MW cache
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Tarrow
	Nov 21 2024, 9:03 AM

Description

We wish to try to see if our redis session expiry rate can be reduced by moving the highly volatile data away from the relatively stable data.

We will deploy a new cluster for this (using argo). And then configure it to be used for the main MediaWiki cache as per the comment at T376853#10234021

Charts change https://github.com/wbstack/charts/pull/178
Staging https://github.com/wmde/wbaas-deploy/commit/ea7c027a63699e029c7cb274a9363fa30a396b85#diff-f64d247983d17f7ce876d21cf69063b9f25d737b15d517cf7c72ddf1bdc64c19
To fit on staging https://github.com/wmde/wbaas-deploy/pull/1869
downgrading chart for redis license https://github.com/wmde/wbaas-deploy/pull/1873
MW change to use new env vars https://github.com/wbstack/mediawiki/pull/461
chart change for MW new env vars https://github.com/wbstack/charts/pull/180

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Anton.Kokh	T376853 🤖 Issues with sessions on wikibase.cloud?
		Resolved		Tarrow	T380448 🤖 Deploy new redis cluster for main MW cache

Event Timeline

Tarrow created this task.Nov 21 2024, 9:03 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 21 2024, 9:03 AM

~~Waiting on new redis password~~

Tarrow added a parent task: T376853: 🤖 Issues with sessions on wikibase.cloud?.Nov 25 2024, 5:36 PM

Tarrow renamed this task from Deploy new redis cluster to attempt to reduce session expiry rate to Deploy new redis cluster for main MW cache.Nov 25 2024, 5:39 PM

Tarrow updated the task description. (Show Details)

Tarrow updated the task description. (Show Details)Nov 25 2024, 8:51 PM

Tarrow mentioned this in T376853: 🤖 Issues with sessions on wikibase.cloud?.Nov 25 2024, 9:21 PM

For now it was decided to reuse the other redis cluster password. This has been deployed to staging but unfortunately won't quite "fit"

Warning  FailedScheduling   27s (x104 over 8h)     default-scheduler   0/3 nodes are available: 1 Insufficient memory, 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Tarrow assigned this task to Andrew-WMDE.Nov 25 2024, 9:27 PM

Tarrow updated the task description. (Show Details)

Tarrow mentioned this in T380795: Set separate password for `redis-2` cluster.Nov 25 2024, 9:31 PM

Tarrow claimed this task.Nov 27 2024, 9:05 AM

Tarrow added a subscriber: Andrew-WMDE.

Tarrow renamed this task from Deploy new redis cluster for main MW cache to 🤖 Deploy new redis cluster for main MW cache.Nov 27 2024, 11:15 AM

Tarrow updated the task description. (Show Details)Nov 27 2024, 3:01 PM

Tarrow updated the task description. (Show Details)Nov 27 2024, 3:14 PM

Tarrow updated the task description. (Show Details)Nov 28 2024, 4:46 PM

Shipped the new version of MW to staging and floundered about for a couple of hours wondering why I couldn't see any keys turning up in redis2. I was checking with KEYS * in redis-cli

I then talked to @Deniz_WMDE and discovered that we hadn't set the password :facepalm: . See this PR for the fix. Shipped to staging and it was behaving well with 91 keys immediately appearing in the new cluster.

Now shipping the new argocd-config chart to production and intend to followup with the new mediawiki chart and image

Tarrow moved this task from In Review to Done on the Wikibase Cloud (Kanban Board Q4 2024) board.Nov 28 2024, 4:58 PM

Anton.Kokh closed this task as Resolved.Mon, Dec 2, 3:47 PM

🤖 Deploy new redis cluster for main MW cacheClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

🤖 Deploy new redis cluster for main MW cache
Closed, ResolvedPublic
Actions

Related Objects
Search...