Page MenuHomePhabricator
Create Task
Maniphest T346657

Requests originating from zhwiki wikifeeds caused parsoid outage
Open, MediumPublic
Actions

Description

From ops mailing list:

There was a restbase/parsoid outage due to a flood of requests for transforming pages on zhwiki.
This was eventually traced back to requests to the /api/rest_v1/feed/featured/yyyy/mm/dd URLs, which seem to cascade (via restbase, mobileapps, restbase again) into making 10+ requests to parsoid to urls like:

http://zh.wikipedia.org/w/rest.php/zh.wikipedia.org/v3/transform/pagebundle/to/pagebundle/<title>

These requests are expensive to parse causing the outage.

NOTE: This wasn't the first time we saw such an outage, but it was the first time we could identify the origin thanks to moving wikifeeds out of restbase and to the api gateway, with its superior observability stack.

Here is the grafana diagram for parsoid performance where the spike on language conversion is visible.
https://grafana.wikimedia.org/goto/X77YIsmIz?orgId=1

Details

ProjectBranchLines +/-Subject
mediawiki/services/parsoidmaster+8 -1Disable Zh Language converter
mediawiki/services/wikifeedsmaster+8 -1Strip zh variants when calling summary
mediawiki/services/wikifeedsmaster+9 -1Strip zh variants when calling parsoid
mediawiki/services/parsoidmaster+1 -1Disable Zh Language converter
Customize query in gerrit

Related Objects

Event Timeline

Jgiannelos created this task.Mon, Sep 18, 3:44 PM
Restricted Application added subscribers: Stang, Aklapper. · View Herald TranscriptMon, Sep 18, 3:44 PM
Jgiannelos updated the task description. (Show Details)Mon, Sep 18, 3:45 PM
Joe added a subscriber: Joe.Mon, Sep 18, 3:48 PM
Jgiannelos added a comment.Mon, Sep 18, 4:32 PM

From my local restbase setup I only end-up querying pagebundle/to/bundle when I am passing a specific locale to /page/html.
For example:

This doesn't hit the transformation that is computationally intensive:

curl -v 127.0.0.1:7233/zh.wikipedia.beta.wmflabs.org/v1/page/html/%E5%8D%97%E5%8C%97%E6%9C%9D

but this does:

curl -v 127.0.0.1:7233/zh.wikipedia.beta.wmflabs.org/v1/page/html/%E5%8D%97%E5%8C%97%E6%9C%9D -H 'accept-language: zh-cn'

When it comes to wikifeeds, the backend GETs page/html endpoint with the locale passed from the clients.

gerritbot added a comment.Mon, Sep 18, 9:21 PM

Change 958573 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] Disable Zh Language converter

https://gerrit.wikimedia.org/r/958573

gerritbot added a project: Patch-For-Review.Mon, Sep 18, 9:21 PM
gerritbot added a comment.Tue, Sep 19, 2:30 AM

Change 958573 abandoned by Subramanya Sastry:

[mediawiki/services/parsoid@master] Disable Zh Language converter

Reason:

I will go with Scott's suggestion here.

https://gerrit.wikimedia.org/r/958573

gerritbot added a comment.Tue, Sep 19, 2:31 AM

Change 958593 had a related patch set uploaded (by Subramanya Sastry; author: Subramanya Sastry):

[mediawiki/services/parsoid@master] Disable Zh Language converter

https://gerrit.wikimedia.org/r/958593

Shizhao added a project: Chinese-Sites.Tue, Sep 19, 2:40 AM
Shizhao moved this task from Backlog to Apps/Tools/Services on the Chinese-Sites board.
Jgiannelos added a project: Content-Transform-Team-WIP.Tue, Sep 19, 2:54 PM
Jgiannelos moved this task from Backlog to In Progress on the Content-Transform-Team-WIP board.
gerritbot added a comment.Tue, Sep 19, 4:20 PM

Change 958995 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/wikifeeds@master] Strip zh variants when calling parsoid

https://gerrit.wikimedia.org/r/958995

gerritbot added a comment.Tue, Sep 19, 6:43 PM

Change 958995 merged by jenkins-bot:

[mediawiki/services/wikifeeds@master] Strip zh variants when calling parsoid

https://gerrit.wikimedia.org/r/958995

Jgiannelos mentioned this in rWSWFbfc6f19de9fa: Strip zh variants when calling parsoid.Tue, Sep 19, 8:37 PM
Jgiannelos moved this task from In Progress to To Verify on the Content-Transform-Team-WIP board.Wed, Sep 20, 3:08 PM
Jgiannelos added a comment.EditedWed, Sep 20, 3:13 PM

There is a workaround for wikifeeds that should fix the parsoid outage issue in the short term. Can we re-enable traffic for zhwiki on wikifeeds?

Jgiannelos added a project: serviceops.Wed, Sep 20, 3:13 PM
akosiaris added a subscriber: akosiaris.Thu, Sep 21, 8:22 AM

I 've just disabled the rule. It's still present, but inactive. For other SREs having to re-enable it in an emergency:

puppetmaster1001:$ sudo requestctl enable cache-text/wikifeeds_featured
puppetmaster1001:$ sudo requestctl commit
MSantos triaged this task as Medium priority.Thu, Sep 21, 2:17 PM
MSantos removed projects: Content-Transform-Team, Wikifeeds.
akosiaris added a comment.Thu, Sep 21, 2:45 PM

I 've just re-enabled the filter, rejecting traffic, we are meeting issues with high latencies and decreased availability in the parsoid cluster.

gerritbot added a comment.Thu, Sep 21, 5:35 PM

Change 959822 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/wikifeeds@master] Strip zh variants when calling summary

https://gerrit.wikimedia.org/r/959822

Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL