Page MenuHomePhabricator
Create Task
Maniphest T334232

Wikitech experiencing a spike of stale cache errors since 2023-03-15
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Several tickets are reporting a similar failure cause of mismatch of cache state and database state for user/preference data:

https://logstash.wikimedia.org/goto/c90d1a3ada8d842ab1dcb6a6b4dc111d shows a trending pattern of "CAS update failed" errors for wikitech:

Screen Shot 2023-04-06 at 9.12.36 AM.png (692×1 px, 39 KB)

Details

SubjectRepoBranchLines +/-
mc: Fix accidental mcrouter prefix $wgWANObjectCache on labswikioperations/mediawiki-configmaster+5 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

bd808 created this task.Apr 6 2023, 3:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 6 2023, 3:14 PM
RhinosF1 subscribed.Apr 6 2023, 3:19 PM
bd808 added subtasks: T333925: Error during MFA setup for wikitech.wikimedia.org: MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version., T334102: Wikitech: Preferences not updating after email change.Apr 6 2023, 3:28 PM
bd808 changed the subtype of this task from "Bug Report" to "Production Error".
thcipriani moved this task from Untriaged to Apr 2023 on the Wikimedia-production-error board.Apr 13 2023, 3:48 PM
jcrespo subscribed.Apr 18 2023, 2:32 PM

maybe related? T334487

bd808 added a subtask: T335456: Cache issues preventing Wikitech login after attempting to add new pubkey.Apr 26 2023, 10:16 PM
bd808 mentioned this in T335456: Cache issues preventing Wikitech login after attempting to add new pubkey.
gerritbot added a comment.Apr 26 2023, 11:59 PM

Change 912421 had a related patch set uploaded (by Krinkle; author: Krinkle):

[operations/mediawiki-config@master] mc: Fix accidental mcrouter prefix $wgWANObjectCache on labswiki

https://gerrit.wikimedia.org/r/912421

gerritbot added a project: Patch-For-Review.Apr 26 2023, 11:59 PM
gerritbot added a comment.Apr 27 2023, 3:20 PM

Change 912421 merged by jenkins-bot:

[operations/mediawiki-config@master] mc: Fix accidental mcrouter prefix $wgWANObjectCache on labswiki

https://gerrit.wikimedia.org/r/912421

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2023, 3:30 PM
bd808 closed subtask T334102: Wikitech: Preferences not updating after email change as Resolved.Apr 27 2023, 5:44 PM
bd808 mentioned this in T334102: Wikitech: Preferences not updating after email change.
bd808 closed this task as Resolved.Apr 27 2023, 6:10 PM
bd808 assigned this task to Krinkle.
bd808 added a subscriber: Krinkle.

I'm strongly confident that @Krinkle's config change has resolved the spike. We should wait for confirmation on individual sub-tasks before closing as they may still have unique issues that just looked like an example of the collective problem.

Aklapper closed subtask T333925: Error during MFA setup for wikitech.wikimedia.org: MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version. as Resolved.Jul 2 2023, 11:27 AM
bd808 closed subtask T335456: Cache issues preventing Wikitech login after attempting to add new pubkey as Resolved.Jul 10 2023, 3:17 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL