Page MenuHomePhabricator
Create Task
Maniphest T249595

Purge / Reject client pages that were cached in parser cache during the T249565 (wb_items_per_site) incident
Closed, ResolvedPublic
Actions

Description

Rather than actively purge we probably want to use https://www.mediawiki.org/wiki/Manual:Hooks/RejectParserCacheValue

Monitoring of the rejection rate will be possible at https://grafana.wikimedia.org/d/000000106/parser-cache

Start Time: 23:00:00 UTC 6 April 2019
End Time: TBA (The EndTime should be once T249596: Rebuild wb_items_per_site, after incident where wb_items_per_site was dropped has been completed and the table data is restored)

The time period that we are rejecting for will be rolled out in increments to ensure that we don't have to reparse too many pages at once.
Starting with a 1 hour time period and evaluating the time period after the initial deployment.

Details

ProjectBranchLines +/-Subject
operations/mediawiki-configmaster+2 -3RejectParserCache entries for wb_items_per_site 14.5/14.5 hours
operations/mediawiki-configmaster+1 -1RejectParserCache entries for wb_items_per_site 12/14.5 hours
operations/mediawiki-configmaster+1 -1RejectParserCache entries for wb_items_per_site 10/14.5 hours
operations/mediawiki-configmaster+1 -1RejectParserCache entries for wb_items_per_site 8/14.5 hours
operations/mediawiki-configmaster+1 -1RejectParserCache entries for wb_items_per_site 4/14.5 hours
operations/mediawiki-configmaster+1 -1RejectParserCache entries for wb_items_per_site 2/14.5 hours
operations/mediawiki-configmaster+0 -0RejectParserCacheValue entries during wb_items_per_site drop incident: namespace check
operations/mediawiki-configmaster+21 -0RejectParserCacheValue entries during wb_items_per_site drop incident 2
operations/mediawiki-configmaster+24 -0RejectParserCacheValue entries during wb_items_per_site drop incident
Customize query in gerrit

Related Objects
Search...

Event Timeline

Addshore created this task.Apr 7 2020, 8:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 7 2020, 8:40 AM
Addshore updated the task description. (Show Details)Apr 7 2020, 8:51 AM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Apr 7 2020, 9:15 AM
Joe added a subscriber: Joe.Apr 7 2020, 9:23 AM

I think the right way to do this would be to emit an htmlCacheUpdate job for every wikidata edit in the interval.

These will:

  • recursively find all linked pages (not sure this works for wikibase items though - you might know that better)
  • invalidate their parsercache
  • send a purge to the CDN

Anyways, I would advise against doing this, and just invalidate the parser cache only, and let the CDN cache just expire by itself, as we're already under an excessive purge pressure (see T249325)

Addshore added a comment.Apr 7 2020, 9:24 AM
In T249595#6035565, @Joe wrote:

Anyways, I would advise against doing this, and just invalidate the parser cache only, and let the CDN cache just expire by itself, as we're already under an excessive purge pressure (see T249325)

Sounds good, I'll just do that!

Addshore updated the task description. (Show Details)Apr 7 2020, 9:24 AM
gerritbot added a comment.Apr 7 2020, 11:06 AM

Change 587231 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident

https://gerrit.wikimedia.org/r/587231

gerritbot added a project: Patch-For-Review.Apr 7 2020, 11:06 AM
Addshore updated the task description. (Show Details)Apr 7 2020, 11:15 AM
• zeljkofilipin triaged this task as Unbreak Now! priority.Apr 7 2020, 11:32 AM
• zeljkofilipin added a subscriber: • zeljkofilipin.

Train blockers are UBN.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptApr 7 2020, 11:32 AM
• zeljkofilipin removed a subscriber: • zeljkofilipin.Apr 7 2020, 11:33 AM
Addshore mentioned this in T249565: Wikidata's wb_items_per_site table has suddenly disappeared, creating DBQueryErrors on page views.Apr 7 2020, 12:22 PM
Sakretsu added a subscriber: Sakretsu.Apr 7 2020, 12:25 PM
gerritbot added a comment.Apr 7 2020, 1:50 PM

Change 587231 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident

https://gerrit.wikimedia.org/r/587231

Stashbot added a comment.Apr 7 2020, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T13:55:21Z] <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 29s)

Stashbot added a comment.Apr 7 2020, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T13:57:04Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: REVERT T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 58s)

gerritbot added a comment.Apr 7 2020, 2:03 PM

Change 587264 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident 2

https://gerrit.wikimedia.org/r/587264

Addshore added a comment.Apr 7 2020, 2:05 PM

2:57 PM <addshore> Error from line 3467 of /srv/mediawiki/wmf-config/CommonSettings.php: Function name must be a string
2:58 PM <hoo> Ah, call_user_func?!
2:59 PM <addshore> hoo: not actually sure thogh, $wgWBClientSettings['excludeNamespaces']() works for me
3:00 PM <hoo> Maybe it's not set in some weird condition (or already resolved… is Wikibase storing the resolved value back into the array?)
3:00 PM <addshore> hmmmmmmmmmmm
3:01 PM <addshore> hoo: I might just remove and continue without the namespace check
3:01 PM <addshore> I dont think we will cause too many unneeded re parses in the grand scheme of things
3:01 PM <hoo> Probably

gerritbot added a comment.Apr 7 2020, 2:06 PM

Change 587264 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident 2

https://gerrit.wikimedia.org/r/587264

Stashbot added a comment.Apr 7 2020, 2:08 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:08:03Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) take 2 (duration: 00m 57s)

gerritbot added a comment.Apr 7 2020, 2:09 PM

Change 587267 had a related patch set uploaded (by DannyS712; owner: DannyS712):
[operations/mediawiki-config@master] RejectParserCacheValue entries during wb_items_per_site drop incident: namespace check

https://gerrit.wikimedia.org/r/587267

gerritbot added a comment.Apr 7 2020, 2:11 PM

Change 587267 abandoned by DannyS712:
RejectParserCacheValue entries during wb_items_per_site drop incident: namespace check

https://gerrit.wikimedia.org/r/587267

gerritbot added a comment.Apr 7 2020, 2:12 PM

Change 587268 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 2/14.5 hours

https://gerrit.wikimedia.org/r/587268

gerritbot added a comment.Apr 7 2020, 2:14 PM

Change 587268 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 2/14.5 hours

https://gerrit.wikimedia.org/r/587268

Stashbot added a comment.Apr 7 2020, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:15:52Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (2/14.5h) (duration: 00m 58s)

gerritbot added a comment.Apr 7 2020, 2:22 PM

Change 587271 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 4/14.5 hours

https://gerrit.wikimedia.org/r/587271

gerritbot added a comment.Apr 7 2020, 2:23 PM

Change 587271 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 4/14.5 hours

https://gerrit.wikimedia.org/r/587271

Stashbot added a comment.Apr 7 2020, 2:25 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:25:34Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (4/14.5h) (duration: 00m 58s)

gerritbot added a comment.Apr 7 2020, 2:30 PM

Change 587273 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 8/14.5 hours

https://gerrit.wikimedia.org/r/587273

gerritbot added a comment.Apr 7 2020, 2:31 PM

Change 587273 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 8/14.5 hours

https://gerrit.wikimedia.org/r/587273

Stashbot added a comment.Apr 7 2020, 2:35 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:35:02Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (8/14.5h) (duration: 00m 58s)

Addshore added a comment.Apr 7 2020, 2:52 PM

Purging well under way

image.png (320×920 px, 31 KB)

gerritbot added a comment.Apr 7 2020, 2:52 PM

Change 587277 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 10/14.5 hours

https://gerrit.wikimedia.org/r/587277

gerritbot added a comment.Apr 7 2020, 2:53 PM

Change 587277 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 10/14.5 hours

https://gerrit.wikimedia.org/r/587277

Stashbot added a comment.Apr 7 2020, 2:56 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T14:56:27Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (10/14.5h) (duration: 00m 55s)

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2020, 3:10 PM
gerritbot added a comment.Apr 7 2020, 3:14 PM

Change 587280 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 12/14.5 hours

https://gerrit.wikimedia.org/r/587280

gerritbot added a project: Patch-For-Review.Apr 7 2020, 3:14 PM
gerritbot added a comment.Apr 7 2020, 3:16 PM

Change 587280 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 12/14.5 hours

https://gerrit.wikimedia.org/r/587280

Stashbot added a comment.Apr 7 2020, 3:17 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T15:17:52Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (12/14.5h) (duration: 01m 00s)

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2020, 4:10 PM
gerritbot added a comment.Apr 7 2020, 5:50 PM

Change 587303 had a related patch set uploaded (by Addshore; owner: Addshore):
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 14.5/14.5 hours

https://gerrit.wikimedia.org/r/587303

gerritbot added a project: Patch-For-Review.Apr 7 2020, 5:50 PM
Addshore lowered the priority of this task from Unbreak Now! to High.Apr 7 2020, 5:52 PM

No longer a train blocker

gerritbot added a comment.Apr 7 2020, 5:52 PM

Change 587303 merged by jenkins-bot:
[operations/mediawiki-config@master] RejectParserCache entries for wb_items_per_site 14.5/14.5 hours

https://gerrit.wikimedia.org/r/587303

Stashbot added a comment.Apr 7 2020, 5:54 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T17:54:03Z] <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) (duration: 01m 16s)

Stashbot added a comment.Apr 7 2020, 5:55 PM

Mentioned in SAL (#wikimedia-operations) [2020-04-07T17:55:21Z] <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) retry (duration: 01m 02s)

Addshore closed this task as Resolved.Apr 7 2020, 5:55 PM

All pcache entries that were created while the tables were broken will now be rejected.

Addshore added a comment.Apr 7 2020, 6:04 PM

image.png (724×923 px, 99 KB)

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2020, 6:10 PM
Rowingbohe added a subscriber: Rowingbohe.Apr 12 2020, 3:28 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL