Page MenuHomePhabricator

Purge / Reject client pages that were cached in parser cache during the T249565 (wb_items_per_site) incident
Closed, ResolvedPublic

Description

Rather than actively purge we probably want to use https://www.mediawiki.org/wiki/Manual:Hooks/RejectParserCacheValue

Monitoring of the rejection rate will be possible at https://grafana.wikimedia.org/d/000000106/parser-cache

Start Time: 23:00:00 UTC 6 April 2019
End Time: TBA (The EndTime should be once T249596: Rebuild wb_items_per_site, after incident where wb_items_per_site was dropped has been completed and the table data is restored)

The time period that we are rejecting for will be rolled out in increments to ensure that we don't have to reparse too many pages at once.
Starting with a 1 hour time period and evaluating the time period after the initial deployment.

Event Timeline

Addshore created this task.Apr 7 2020, 8:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 7 2020, 8:40 AM
Addshore updated the task description. (Show Details)Apr 7 2020, 8:51 AM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Apr 7 2020, 9:15 AM
Joe added a subscriber: Joe.Apr 7 2020, 9:23 AM

I think the right way to do this would be to emit an htmlCacheUpdate job for every wikidata edit in the interval.

These will:

  • recursively find all linked pages (not sure this works for wikibase items though - you might know that better)
  • invalidate their parsercache
  • send a purge to the CDN

Anyways, I would advise against doing this, and just invalidate the parser cache only, and let the CDN cache just expire by itself, as we're already under an excessive purge pressure (see T249325)

Anyways, I would advise against doing this, and just invalidate the parser cache only, and let the CDN cache just expire by itself, as we're already under an excessive purge pressure (see T249325)

Sounds good, I'll just do that!

Addshore updated the task description. (Show Details)Apr 7 2020, 9:24 AM

Change 587231 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident

https://gerrit.wikimedia.org/r/587231

Addshore updated the task description. (Show Details)Apr 7 2020, 11:15 AM
zeljkofilipin triaged this task as Unbreak Now! priority.Apr 7 2020, 11:32 AM
zeljkofilipin added a subscriber: zeljkofilipin.

Train blockers are UBN.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptApr 7 2020, 11:32 AM

Change 587231 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident

https://gerrit.wikimedia.org/r/587231

Mentioned in SAL (#wikimedia-operations) [2020-04-07T13:55:21Z] <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 29s)

Mentioned in SAL (#wikimedia-operations) [2020-04-07T13:57:04Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: REVERT T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 58s)

Change 587264 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident 2

https://gerrit.wikimedia.org/r/587264

2:57 PM <addshore> Error from line 3467 of /srv/mediawiki/wmf-config/CommonSettings.php: Function name must be a string
2:58 PM <hoo> Ah, call_user_func?!
2:59 PM <addshore> hoo: not actually sure thogh, $wgWBClientSettings['excludeNamespaces']() works for me
3:00 PM <hoo> Maybe it's not set in some weird condition (or already resolved… is Wikibase storing the resolved value back into the array?)
3:00 PM <addshore> hmmmmmmmmmmm
3:01 PM <addshore> hoo: I might just remove and continue without the namespace check
3:01 PM <addshore> I dont think we will cause too many unneeded re parses in the grand scheme of things
3:01 PM <hoo> Probably

Change 587264 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident 2

https://gerrit.wikimedia.org/r/587264

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:08:03Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) take 2 (duration: 00m 57s)

Change 587267 had a related patch set uploaded (by DannyS712; owner: DannyS712):
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident: namespace check

https://gerrit.wikimedia.org/r/587267

Change 587267 abandoned by DannyS712:
RejectParserCacheValue entries during wb_items_per_site drop incident: namespace check

https://gerrit.wikimedia.org/r/587267

Change 587268 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 2/14.5 hours

https://gerrit.wikimedia.org/r/587268

Change 587268 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 2/14.5 hours

https://gerrit.wikimedia.org/r/587268

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:15:52Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (2/14.5h) (duration: 00m 58s)

Change 587271 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 4/14.5 hours

https://gerrit.wikimedia.org/r/587271

Change 587271 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 4/14.5 hours

https://gerrit.wikimedia.org/r/587271

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:25:34Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (4/14.5h) (duration: 00m 58s)

Change 587273 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 8/14.5 hours

https://gerrit.wikimedia.org/r/587273

Change 587273 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 8/14.5 hours

https://gerrit.wikimedia.org/r/587273

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:35:02Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (8/14.5h) (duration: 00m 58s)

Purging well under way

Change 587277 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 10/14.5 hours

https://gerrit.wikimedia.org/r/587277

Change 587277 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 10/14.5 hours

https://gerrit.wikimedia.org/r/587277

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:56:27Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (10/14.5h) (duration: 00m 55s)

Change 587280 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 12/14.5 hours

https://gerrit.wikimedia.org/r/587280

Change 587280 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 12/14.5 hours

https://gerrit.wikimedia.org/r/587280

Mentioned in SAL (#wikimedia-operations) [2020-04-07T15:17:52Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (12/14.5h) (duration: 01m 00s)

Change 587303 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 14.5/14.5 hours

https://gerrit.wikimedia.org/r/587303

Addshore lowered the priority of this task from Unbreak Now! to High.Apr 7 2020, 5:52 PM

No longer a train blocker

Change 587303 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 14.5/14.5 hours

https://gerrit.wikimedia.org/r/587303

Mentioned in SAL (#wikimedia-operations) [2020-04-07T17:54:03Z] <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) (duration: 01m 16s)

Mentioned in SAL (#wikimedia-operations) [2020-04-07T17:55:21Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) retry (duration: 01m 02s)

Addshore closed this task as Resolved.Apr 7 2020, 5:55 PM

All pcache entries that were created while the tables were broken will now be rejected.