Maniphest T220574

Continuous errors on several REST API resources (probably related to MCS release)
Closed, ResolvedPublic
Actions

Description

Right after MCS was released yesterday (per SAL)

17:15 	<bsitzmann@deploy1001> 	Started deploy [mobileapps/deploy@b04c397]: Update mobileapps to 3edfcad (T220045 T219411 T219667)

we've seen a sudden increase in the number of errors reported by our edge servers. Upon inspecting the issue this morning, I found that indeed some urls return errors consistently, see for instance

https://fr.wikipedia.org/api/rest_v1/page/media/Nikolai_Gorbachev

this must be fixed ASAP, I am even tempted to do a MCS rollback right now, but I am unsure what that would mean for anything relying on it.

Event Timeline

Joe created this task.Apr 10 2019, 6:23 AM

Restricted Application added a project: Product-Infrastructure-Team-Backlog. · View Herald TranscriptApr 10 2019, 6:23 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Joe triaged this task as Unbreak Now! priority.Apr 10 2019, 6:23 AM

Restricted Application added subscribers: Liuxinyu970226, Mholloway, TerraCodes. · View Herald TranscriptApr 10 2019, 6:23 AM

The errors seem to come from restbase: https://logstash.wikimedia.org/app/kibana#/dashboard/restbase?_g=h@1251ff0&_a=h@281d689

I've looked around a bit, and while the number of errors is in general below the SLO we have for restbase:

they all come from URLs pertaining to MCS AFAICS
some resources consistently do not render

So I would consider this a UBN! ticket even if the error rate is not that significant.

Change 502686 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/mobileapps@master] Revert "Bifurcate imageinfo queries to improve performance"

https://gerrit.wikimedia.org/r/502686

gerritbot added a project: Patch-For-Review.Apr 10 2019, 6:59 AM

Mholloway claimed this task.Apr 10 2019, 7:03 AM

Mholloway edited projects, added Product-Infrastructure-Team-Backlog (Kanban); removed Product-Infrastructure-Team-Backlog.

Mholloway moved this task from To Do to Code Review on the Product-Infrastructure-Team-Backlog (Kanban) board.

Mholloway added a subscriber: bearND.

Change 502686 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Revert "Bifurcate imageinfo queries to improve performance"

https://gerrit.wikimedia.org/r/502686

Mentioned in SAL (#wikimedia-operations) [2019-04-10T07:18:40Z] <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@efd5bd5]: Revert "Bifurcate imageinfo queries to improve performance" (T220574)

Mentioned in SAL (#wikimedia-operations) [2019-04-10T07:22:45Z] <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@efd5bd5]: Revert "Bifurcate imageinfo queries to improve performance" (T220574) (duration: 04m 05s)

https://grafana.wikimedia.org/d/myRmf1Pik/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=text&var-status_type=5&from=now-1h&to=now

the deploy solved the issue, thanks @Mholloway for solving this so quickly!

mobrovac edited projects, added Services (done), Core Platform Team Workboards (Done with CPT), Core Platform Team (Needs Cleaning - Security, stability, performance, and scalability (TEC1)); removed Patch-For-Review, Services.Apr 10 2019, 4:42 PM

Continuous errors on several REST API resources (probably related to MCS release)Closed, ResolvedPublicActions

Description

Event Timeline

Continuous errors on several REST API resources (probably related to MCS release)
Closed, ResolvedPublic
Actions