Re-imaged mw app servers can end up with missing l10n cache for old versions of MW needed for rollback
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	Reedy
	Jan 29 2021, 10:34 PM

Description

Seen during the rollback today.

Various re-imaged codfw servers (T245757) are throwing errors about no l10ncache, as they presumably did scap pull when all wikis were already on .28, so .27 wasn't rebuilt on those servers. This means when the train was reverted all back to .27, they had no l10n cache waiting to go

No obvious user facing impact (this time!), but icinga is obviously unhappy. And if happened on app servers in the DC currently serving MW.. It could be a much less happy place

[22:35:36] <Reedy> we should make nostalgiawiki always be a version behind
[22:35:41] <Reedy> then we can be nostalgic for "last week"

Related Objects

Mentioned In: T268524: Upgrade Parsoid servers to buster
Mentioned Here: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10)

Event Timeline

Reedy created this task.Jan 29 2021, 10:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 29 2021, 10:34 PM

taavi subscribed.Jan 29 2021, 10:34 PM

Reedy updated the task description. (Show Details)Jan 29 2021, 10:36 PM

Reedy updated the task description. (Show Details)

Reedy updated the task description. (Show Details)Jan 29 2021, 10:38 PM

Reedy added a project: Release-Engineering-Team.Jan 29 2021, 10:41 PM

thcipriani edited projects, added Release-Engineering-Team (Deployment services), Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)); removed Release-Engineering-Team.Jan 29 2021, 11:01 PM

thcipriani moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)) board.

thcipriani added a project: Scap.

It sounds like sync wikiversions should check to make sure the l10n cache exists on each appserver for any not-already-in-use versions being added before actually syncing.

And on the reimage side, if there could be a scap command that would rebuild the l10n cache for N-1 version in case of a rollback, we can run that after reimaging.

Legoktm triaged this task as High priority.Jan 30 2021, 9:10 AM

Legoktm added a parent task: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).

Ladsgroup subscribed.Jan 30 2021, 9:14 AM

Aklapper renamed this task from Re-imaged mw app servers can end up with missing l10n cache for old versons of MW needed for rollback to Re-imaged mw app servers can end up with missing l10n cache for old versions of MW needed for rollback.Jan 30 2021, 11:21 AM

• brennen subscribed.Feb 9 2021, 11:51 PM

This happened again yesterday during the rollback.

RhinosF1 subscribed.Feb 13 2021, 12:50 PM

Dzahn mentioned this in T268524: Upgrade Parsoid servers to buster .Apr 2 2021, 10:16 PM

thcipriani edited projects, added Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)); removed Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)).Apr 20 2021, 12:37 AM

thcipriani removed a project: Release-Engineering-Team (Deployment services).Apr 20 2021, 1:10 AM

thcipriani edited projects, added Release-Engineering-Team (Doing); removed Release-Engineering-Team-TODO (2021-04-01 to 2021-06-30 (Q4)).Apr 20 2021, 4:25 AM

thcipriani edited projects, added Release-Engineering-Team (Next); removed Release-Engineering-Team (Doing).May 4 2021, 4:45 PM

We proceeded with the wider work without fixing this task, so I'll remove it as a blocker.

Jdforrester-WMF removed a parent task: T245757: Upgrade MediaWiki clusters to Debian Buster (debian 10).Nov 30 2021, 8:20 PM

thcipriani edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team (Next).Sep 7 2022, 3:46 PM

jbond edited projects, added serviceops; removed SRE.Nov 4 2022, 1:32 PM

jijiki moved this task from Incoming 🐫 to 🍦IceBox on the serviceops board.Nov 14 2022, 1:21 PM

The move to mw-on-k8s should obviate the need for this work.

Re-imaged mw app servers can end up with missing l10n cache for old versions of MW needed for rollbackClosed, DeclinedPublicActions

Description

Related Objects

Event Timeline

Re-imaged mw app servers can end up with missing l10n cache for old versions of MW needed for rollback
Closed, DeclinedPublic
Actions