Change Details

From ops mailing list: There was a restbase/parsoid outage due to a flood of requests for transforming pages on zhwiki. This was eventually traced back to requests to the /api/rest_v1/feed/featured/yyyy/mm/dd URLs, which seem to cascade (via restbase, mobileapps, restbase again) into making 10+ requests to parsoid to urls like: http://zh.wikipedia.org/w/rest.php/zh.wikipedia.org/v3/transform/pagebundle/to/pagebundle/<title> These requests are expensive to parse causing the outage. NOTE: This wasn't the first time we saw such an outage, but it was the first time we could identify the origin thanks to moving wikifeeds out of restbase and to the api gateway, with its superior observability stack. Here is the grafana diagram for parsoid performance where the spike on language conversion is visible. https://grafana.wikimedia.org/goto/X77YIsmIz?orgId=1

From ops mailing list: There was a restbase/parsoid outage due to a flood of requests for transforming pages on zhwiki. This was eventually traced back to requests to the `/api/rest_v1/feed/featured/yyyy/mm/dd` URLs, which seem to cascade (via restbase, mobileapps, restbase again) into making 10+ requests to parsoid to urls like: ``` http://zh.wikipedia.org/w/rest.php/zh.wikipedia.org/v3/transform/pagebundle/to/pagebundle/<title> ``` These requests are expensive to parse causing the outage. NOTE: This wasn't the first time we saw such an outage, but it was the first time we could identify the origin thanks to moving wikifeeds out of restbase and to the api gateway, with its superior observability stack. Here is the grafana diagram for parsoid performance where the spike on language conversion is visible. https://grafana.wikimedia.org/goto/X77YIsmIz?orgId=1

From ops mailing list: There was a restbase/parsoid outage due to a flood of requests for transforming pages on zhwiki. This was eventually traced back to requests to the `/api/rest_v1/feed/featured/yyyy/mm/dd` URLs, which seem to cascade (via restbase, mobileapps, restbase again) into making 10+ requests to parsoid to urls like: ``` http://zh.wikipedia.org/w/rest.php/zh.wikipedia.org/v3/transform/pagebundle/to/pagebundle/<title> ``` These requests are expensive to parse causing the outage. NOTE: This wasn't the first time we saw such an outage, but it was the first time we could identify the origin thanks to moving wikifeeds out of restbase and to the api gateway, with its superior observability stack. Here is the grafana diagram for parsoid performance where the spike on language conversion is visible. https://grafana.wikimedia.org/goto/X77YIsmIz?orgId=1