Page MenuHomePhabricator
Create Task
Maniphest T258186

Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/site" endpoint takes way longer than on scb to complete
Closed, ResolvedPublic
Actions

Description

We got the following alert

PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/data/css/mobile/site (Get site-specific CSS) timed out before a response was received https://wikitech.wikimedia.org/wiki/Mobileapps_%28service%29

Which is consistent with https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?panelId=20&fullscreen&orgId=1&refresh=5m&from=now-1h&to=now&var-dc=codfw%20prometheus%2Fk8s&var-service=mobileapps&var-container_name=All having a suspiciously flat line for --domain_v1_data_css_mobile_site. Quantiles aren't much better (https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?panelId=37&fullscreen&orgId=1&refresh=5m&from=now-1h&to=now&var-dc=codfw%20prometheus%2Fk8s&var-service=mobileapps&var-container_name=All). Again suspiciously flat.

This is indicative of a timeout of some sort, however it's still unclear why this is happening, the migration was paused and the percentage of traffic reaching k8s dropped back to 10%. Even if those requests fail, restbase will retry and probably sent to an scb host which will reply and then the response will be cached.

Details

SubjectRepoBranchLines +/-
Update mobileapps to 2020-07-17-172246-productionoperations/deployment-chartsmaster+3 -3
Add request template for load.phpmediawiki/services/mobileapps/deploymaster+9 -0
Add mw_resource_loader_uri to Node.js service config varsoperations/puppetproduction+1 -0
Site-CSS: Fix load.php request to use request templatesmediawiki/services/mobileappsmaster+77 -23
mobileapps: add request template for load.php requests to configoperations/deployment-chartsmaster+329 -301
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT302086 Set scap minimum python version to 3.7
 ResolvedNoneT247045 Migrate all of production metal and VMs to Buster or later
 ResolvedakosiarisT249724 Track and remove jessie based container images from production
 ResolvedJdforrester-WMFT224908 Drop jessie testing support
 ResolvedJdforrester-WMFT224907 Drop php55 testing support
 ResolvedReedyT205039 Release MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205041 Tracking bug for 1.27.6/1.30.2/1.31.2/1.32.2 security release
 ResolvedBawolffT197279 Direct POST to Special:ChangeEmail will bypass reauth check
 ResolvedAnomieT204729 Potential enwiki DOS due to slow WatchedItemStore::countVisitingWatchersMultiple
 ResolvedBawolffT207603 CVE-2019-12471: Loading JS from user space where the username is not a registered account is dangerous and should be banned
 ResolvedBawolffT208881 CSS using var() to create exponential sized calc() on wiki page will crash visitor's browser
 ResolvedLegoktmT199540 Forbid blocking IP ranges as big as /1 and /2, as done on ruwikiquote using the API
 ResolvedLucas_Werkmeister_WMDET212118 API responses for unpatrolled or (not) autopatrolled recent changes require privileges but may be cached publicly
 ResolvedBawolffT209794 Need to make a limit of count of attempts to change email address
 ResolvedsbassettT222036 Exposed suppressed username or log in Special:EditTags
 ResolvedsbassettT222038 Exposed suppressed log in RevisionDelete page
 ResolvedJdforrester-WMFT221739 Patch jQuery due to CVE-2019-11358
 ResolvedsbassettT25227 Use token when logging out
 ResolvedsbassettT221868 Send out wikitech-l post for T25227 ("Use token when logging out")
 ResolvedReedyT205042 Write and send release announcements for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedReedyT205043 Write and send pre-release Announcements for MediaWiki 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT205044 Update onwiki release notes for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205046 Update HISTORY in master after 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT205047 Tag MW 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedReedyT224499 RELEASE-NOTES for 1.27.6/1.30.2/1.31.2/1.32.2
 ResolvedJdforrester-WMFT224912 Update MediaWiki.org links and versions
 ResolvedJdforrester-WMFT224913 EOL REL1_27 and REL1_30 onwiki
 ResolvedReedyT225201 Formally announce EOL of MW 1.27 and 1.30
 ResolvedReedyT225149 Update CVEs and publish them
 ResolvedReedyT205048 Obtain CVEs for 1.27.6/1.30.2/1.31.2/1.32.2 security releases
 ResolvedJdforrester-WMFT224906 Drop php56 testing support
 ResolvedhasharT223348 Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x
 ResolvedhasharT210287 Migrate wikimedia-fundraising-civicrm to a Docker container
 ResolvedJdforrester-WMFT224905 Move wikiba.se tests from php56 to php72
 ResolvedhasharT224591 Migrate contint* hosts to Buster
 DeclinedMoritzMuehlenhoffT226236 Upload docker-ce 18.06.3 upstream package for Stretch
 InvalidthciprianiT239880 Replacement hardware for buster/stretch upgrade of contint1001 and contint2001
 Resolved• mmodellT215458 Convert zuul to use scap
 ResolvedhasharT240551 Remove Zuul Debian package from WMCS instances
 ResolvedJoeT249110 Build and publish a python2 based container to build wheels
 ResolvedJMeybohmT249812 Rebuild helm/helm-diff for buster-wikimedia
 ResolvedJMeybohmT250479 Rebuild helmfile for buster-wikimedia
 DuplicateDzahnT210008 upgrade krypton (webserver_misc_apps) to stretch
 DeclinedDzahnT224194 switch webserver_misc_apps to PHP 7.2 (7.1)
 ResolvedJdforrester-WMFT211784 Upgrade all CI jobs from node6/npm3 to node10/npm6 across all projects
 OpenNoneT210704 Migrate node-based services in production to node10
 ResolvedakosiarisT198901 Migrate production services to kubernetes using the pipeline
 ResolvedJdforrester-WMFT247012 Archive mediawiki/services/mobileapps/deploy once it's no longer used
 ResolvedakosiarisT218733 Migrate mobileapps to k8s and node 10
 Resolved• MhollowayT258186 Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/site" endpoint takes way longer than on scb to complete

Event Timeline

akosiaris created this task.Jul 16 2020, 3:20 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJul 16 2020, 3:20 PM
• Mholloway added a comment.Jul 16 2020, 3:23 PM

I believe the culprit is this line: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/mobileapps/+/refs/heads/master/lib/css.js#15 which is making a request to the public MediaWiki load.php endpoint rather than requesting from an internal appserver as it should. Will push a fix shortly.

akosiaris triaged this task as High priority.Jul 16 2020, 3:23 PM

I 'll have a look at the networking part. It's quite possible that this is failure to talk to edge caches, as we forbid that in kubernetes.

akosiaris added a comment.Jul 16 2020, 3:24 PM
In T258186#6312650, @Mholloway wrote:

I believe the culprit is this line: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/mobileapps/+/refs/heads/master/lib/css.js#15 which is making a request to the public MediaWiki load.php endpoint rather than requesting from an internal appserver as it should. Will push a fix shortly.

Wow, this was quick! Thanks @Mholloway !

gerritbot added a comment.Jul 16 2020, 5:58 PM

Change 613213 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/mobileapps@master] Site-CSS: Fix load.php request to use request templates

https://gerrit.wikimedia.org/r/613213

gerritbot added a project: Patch-For-Review.Jul 16 2020, 5:58 PM
gerritbot added a comment.Jul 16 2020, 7:04 PM

Change 613237 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/deployment-charts@master] mobileapps: add request template for load.php requests to config

https://gerrit.wikimedia.org/r/613237

• Mholloway moved this task from To Do to Doing on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Jul 16 2020, 7:12 PM
gerritbot added a comment.Jul 17 2020, 7:24 AM

Change 613237 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: add request template for load.php requests to config

https://gerrit.wikimedia.org/r/613237

akosiaris added a comment.Jul 17 2020, 9:17 AM

mobileapps chart change for the load.php request template has been deployed.

gerritbot added a comment.Jul 17 2020, 2:25 PM

Change 613645 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/puppet@production] Add mw_resource_loader_uri to Node.js service config vars

https://gerrit.wikimedia.org/r/613645

gerritbot added a comment.Jul 17 2020, 2:30 PM

Change 613646 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/mobileapps/deploy@master] Add request template for load.php

https://gerrit.wikimedia.org/r/613646

• Mholloway moved this task from Doing to Code Review on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Jul 17 2020, 2:31 PM
gerritbot added a comment.Jul 17 2020, 4:31 PM

Change 613213 merged by jenkins-bot:
[mediawiki/services/mobileapps@master] Site-CSS: Fix load.php request to use request templates

https://gerrit.wikimedia.org/r/613213

gerritbot added a comment.Jul 20 2020, 10:48 AM

Change 613645 merged by Alexandros Kosiaris:
[operations/puppet@production] Add mw_resource_loader_uri to Node.js service config vars

https://gerrit.wikimedia.org/r/613645

gerritbot added a comment.Jul 20 2020, 1:43 PM

Change 613646 merged by jenkins-bot:
[mediawiki/services/mobileapps/deploy@master] Add request template for load.php

https://gerrit.wikimedia.org/r/613646

gerritbot added a comment.Jul 20 2020, 2:29 PM

Change 614754 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/deployment-charts@master] Update mobileapps to 2020-07-17-172246-production

https://gerrit.wikimedia.org/r/614754

gerritbot added a comment.Jul 20 2020, 2:40 PM

Change 614754 merged by jenkins-bot:
[operations/deployment-charts@master] Update mobileapps to 2020-07-17-172246-production

https://gerrit.wikimedia.org/r/614754

• Mholloway closed this task as Resolved.Jul 20 2020, 3:13 PM
• Mholloway moved this task from Code Review to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.

The fix has been rolled out and /data/css/mobile/site response times are now looking much more reasonable (~200-300 ms).

• Mholloway renamed this task from Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/base" endpoint takes way longer than on scb to complete to Investigate why mobileapps in k8s "/{domain}/v1/data/css/mobile/site" endpoint takes way longer than on scb to complete.Jul 20 2020, 3:14 PM
• Mholloway mentioned this in rGMOD702400532320: Add request template for load.php.Jul 20 2020, 3:18 PM
akosiaris awarded a token.Jul 20 2020, 4:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits