Page MenuHomePhabricator
Create Task
Maniphest T346657

Requests originating from zhwiki wikifeeds caused parsoid outage
Closed, ResolvedPublic
Actions

Description

From ops mailing list:

There was a restbase/parsoid outage due to a flood of requests for transforming pages on zhwiki.
This was eventually traced back to requests to the /api/rest_v1/feed/featured/yyyy/mm/dd URLs, which seem to cascade (via restbase, mobileapps, restbase again) into making 10+ requests to parsoid to urls like:

http://zh.wikipedia.org/w/rest.php/zh.wikipedia.org/v3/transform/pagebundle/to/pagebundle/<title>

These requests are expensive to parse causing the outage.

NOTE: This wasn't the first time we saw such an outage, but it was the first time we could identify the origin thanks to moving wikifeeds out of restbase and to the api gateway, with its superior observability stack.

Here is the grafana diagram for parsoid performance where the spike on language conversion is visible.
https://grafana.wikimedia.org/goto/X77YIsmIz?orgId=1

Details

SubjectRepoBranchLines +/-
Use 'variant-metadata-hack' pb2pb for Parsoid-unsupported variantsmediawiki/coremaster+19 -17
Don't call Parsoid's pb2pb to add variant metadatamediawiki/coremaster+29 -23
Disable Zh Language convertermediawiki/services/parsoidmaster+8 -1
Bump wikimedia/parsoid to 0.18.0-a26mediawiki/vendormaster+542 -1 K
Strip zh variants when calling summarymediawiki/services/wikifeedsmaster+8 -1
Strip zh variants when calling parsoidmediawiki/services/wikifeedsmaster+9 -1
Disable Zh Language convertermediawiki/services/parsoidmaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Jgiannelos created this task.Sep 18 2023, 3:44 PM
Restricted Application added subscribers: Stang, Aklapper. · View Herald TranscriptSep 18 2023, 3:44 PM
Jgiannelos updated the task description. (Show Details)Sep 18 2023, 3:45 PM
Joe subscribed.Sep 18 2023, 3:48 PM
Jgiannelos added a comment.Sep 18 2023, 4:32 PM

From my local restbase setup I only end-up querying pagebundle/to/bundle when I am passing a specific locale to /page/html.
For example:

This doesn't hit the transformation that is computationally intensive:

curl -v 127.0.0.1:7233/zh.wikipedia.beta.wmflabs.org/v1/page/html/%E5%8D%97%E5%8C%97%E6%9C%9D

but this does:

curl -v 127.0.0.1:7233/zh.wikipedia.beta.wmflabs.org/v1/page/html/%E5%8D%97%E5%8C%97%E6%9C%9D -H 'accept-language: zh-cn'

When it comes to wikifeeds, the backend GETs page/html endpoint with the locale passed from the clients.

gerritbot added a comment.Sep 18 2023, 9:21 PM

Change 958573 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] Disable Zh Language converter

https://gerrit.wikimedia.org/r/958573

gerritbot added a project: Patch-For-Review.Sep 18 2023, 9:21 PM
gerritbot added a comment.Sep 19 2023, 2:30 AM

Change 958573 abandoned by Subramanya Sastry:

[mediawiki/services/parsoid@master] Disable Zh Language converter

Reason:

I will go with Scott's suggestion here.

https://gerrit.wikimedia.org/r/958573

gerritbot added a comment.Sep 19 2023, 2:31 AM

Change 958593 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] Disable Zh Language converter

https://gerrit.wikimedia.org/r/958593

Shizhao added a project: Chinese-Sites.Sep 19 2023, 2:40 AM
Shizhao moved this task from Backlog to Apps/Tools/Services on the Chinese-Sites board.
Jgiannelos added a project: Content-Transform-Team-WIP.Sep 19 2023, 2:54 PM
Jgiannelos moved this task from Backlog to In Progress on the Content-Transform-Team-WIP board.
gerritbot added a comment.Sep 19 2023, 4:20 PM

Change 958995 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/wikifeeds@master] Strip zh variants when calling parsoid

https://gerrit.wikimedia.org/r/958995

gerritbot added a comment.Sep 19 2023, 6:43 PM

Change 958995 merged by jenkins-bot:

[mediawiki/services/wikifeeds@master] Strip zh variants when calling parsoid

https://gerrit.wikimedia.org/r/958995

Jgiannelos mentioned this in rWSWFbfc6f19de9fa: Strip zh variants when calling parsoid.Sep 19 2023, 8:37 PM
Jgiannelos moved this task from In Progress to To Verify on the Content-Transform-Team-WIP board.Sep 20 2023, 3:08 PM
Jgiannelos added a comment.EditedSep 20 2023, 3:13 PM

There is a workaround for wikifeeds that should fix the parsoid outage issue in the short term. Can we re-enable traffic for zhwiki on wikifeeds?

Jgiannelos added a project: serviceops.Sep 20 2023, 3:13 PM
akosiaris subscribed.Sep 21 2023, 8:22 AM

I 've just disabled the rule. It's still present, but inactive. For other SREs having to re-enable it in an emergency:

puppetmaster1001:$ sudo requestctl enable cache-text/wikifeeds_featured
puppetmaster1001:$ sudo requestctl commit
MSantos triaged this task as Medium priority.Sep 21 2023, 2:17 PM
MSantos removed projects: Content-Transform-Team, Wikifeeds.
akosiaris added a comment.Sep 21 2023, 2:45 PM

I 've just re-enabled the filter, rejecting traffic, we are meeting issues with high latencies and decreased availability in the parsoid cluster.

gerritbot added a comment.Sep 21 2023, 5:35 PM

Change 959822 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/wikifeeds@master] Strip zh variants when calling summary

https://gerrit.wikimedia.org/r/959822

hnowlan subscribed.Sep 25 2023, 9:49 AM
MSantos moved this task from To Verify to To Deploy on the Content-Transform-Team-WIP board.Sep 25 2023, 9:56 AM
gerritbot added a comment.Sep 25 2023, 9:58 AM

Change 959822 merged by jenkins-bot:

[mediawiki/services/wikifeeds@master] Strip zh variants when calling summary

https://gerrit.wikimedia.org/r/959822

Jgiannelos mentioned this in rWSWF45152699cd96: Strip zh variants when calling summary.Sep 25 2023, 10:04 AM
MSantos added a project: Essential-Work.Sep 25 2023, 3:01 PM
LSobanski subscribed.Sep 25 2023, 4:32 PM
gerritbot added a comment.Sep 25 2023, 8:19 PM

Change 958593 merged by jenkins-bot:

[mediawiki/services/parsoid@master] Disable Zh Language converter

https://gerrit.wikimedia.org/r/958593

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 8:31 PM
gerritbot added a comment.Oct 2 2023, 9:11 PM

Change 962706 had a related patch set uploaded (by C. Scott Ananian; author: C. Scott Ananian):

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.18.0-a26

https://gerrit.wikimedia.org/r/962706

gerritbot added a project: Patch-For-Review.Oct 2 2023, 9:11 PM
gerritbot added a comment.Oct 2 2023, 10:28 PM

Change 962706 merged by jenkins-bot:

[mediawiki/vendor@master] Bump wikimedia/parsoid to 0.18.0-a26

https://gerrit.wikimedia.org/r/962706

Maintenance_bot removed a project: Patch-For-Review.Oct 2 2023, 10:31 PM
MSantos moved this task from Needs Triage to Bugs & Crashers on the Parsoid board.Oct 12 2023, 2:44 PM
gerritbot added a comment.Oct 17 2023, 6:18 PM

Change 966593 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/core@master] Use 'variant-metadata-hack' pb2pb for Parsoid-unuspported variants

https://gerrit.wikimedia.org/r/966593

gerritbot added a project: Patch-For-Review.Oct 17 2023, 6:18 PM
Arlolra mentioned this in T349310: DOMException: Invalid Character Error.Oct 19 2023, 11:16 PM
gerritbot added a comment.Oct 20 2023, 8:36 PM

Change 967275 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/core@master] Don't call Parsoid's pb2pb to add variant metadata

https://gerrit.wikimedia.org/r/967275

gerritbot added a comment.Oct 30 2023, 7:25 PM

Change 967275 merged by jenkins-bot:

[mediawiki/core@master] Don't call Parsoid's pb2pb to add variant metadata

https://gerrit.wikimedia.org/r/967275

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.3; 2023-10-31).Oct 30 2023, 8:00 PM
hnowlan mentioned this in T350645: feed/featured endpoint is showing error on zhwiki.Nov 7 2023, 3:52 PM
Stashbot added a comment.Nov 9 2023, 11:38 AM

Mentioned in SAL (#wikimedia-operations) [2023-11-09T11:38:43Z] <_joe_> disabled requestctl cache-text/wikifeeds_featured T350645 T346657

Joe added a comment.Nov 9 2023, 11:44 AM

Given the patch is now live with the latest train, I've disabled the rule for now.

Joe added a comment.Nov 9 2023, 11:46 AM

And good news, most requests to the endpoint now take 50-100ms to get a response, instead than 5-10 seconds. This is much more sustainable.

MSantos moved this task from To Deploy to To Verify on the Content-Transform-Team-WIP board.Nov 13 2023, 4:22 PM
gerritbot added a comment.Nov 27 2023, 4:43 AM

Change 966593 abandoned by Subramanya Sastry:

[mediawiki/core@master] Use 'variant-metadata-hack' pb2pb for Parsoid-unsupported variants

Reason:

https://gerrit.wikimedia.org/r/966593

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2023, 5:10 AM
Jgiannelos closed this task as Resolved.Dec 7 2023, 3:07 PM
Jgiannelos claimed this task.
Stang moved this task from Apps/Tools/Services to Closed on the Chinese-Sites board.Dec 7 2023, 3:31 PM
Stang unsubscribed.Dec 10 2023, 10:33 PM
cooltey mentioned this in T355192: [BUG] Feature article from /feed/featured endpoint is showing incorrect language variant on zhwiki.Apr 2 2024, 9:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL