Page MenuHomePhabricator

App servers still sending handful of profiles XHGui - despite code being disabled
Closed, ResolvedPublic

Description

Despite https://gerrit.wikimedia.org/r/344504 having been deployed over 6 hours ago, I still see a handful of requests per hour to XHGui from app servers. At least from mw1283, mw1220, mw1194, and mw1280.

I verified on each of these servers that the patch is applied there, meaning the code submitting is now in an IF condition only true for X-Wikimedia-Debug requests. From the live php stacktraces I see in logstash I also verified it is indeed coming from /srv/mediawiki/wmf-config/StartProfiler.php line 120 (e.g. not from somewhere else).

This is a problem because tungsten is overloaded and some of the handful of profiles being sent actually end up failing to go through, which then results in an exception being thrown (either fails to connect, or connects but query times out).

It shouldn't affect end-users as far as I know since this all happens in a shutdown function, but still - tungsten is no longer meant to be involved in non-debug requests.

https://logstash.wikimedia.org/goto/02d1871aa9437e67361f182affbadf69

mw1283
WARNING
/w/api.php?format=xml&rawcontinue=1&maxlag=5&action=query&prop=langlinks&...
PHP Warning: failed to connect to 10.64.0.22 [111]: Connection refused

mw1220
ERROR
/w/index.php?title=MediaWiki%3AGadget...&action=raw&..
MongoCursorTimeoutException from line 237 of /srv/mediawiki/multiversion/vendor/mongofill...Timed out waiting for data
`

Details

Related Gerrit Patches:
operations/mediawiki-config : masterStartProfiler: Add hostname in xhgui record

Event Timeline

Krinkle created this task.Mar 24 2017, 7:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2017, 7:35 AM
Krinkle updated the task description. (Show Details)Mar 24 2017, 7:35 AM

Mentioned in SAL (#wikimedia-operations) [2017-03-24T19:33:43Z] <krinkle@tin> Synchronized wmf-config/StartProfiler.php: touch - T161286 - hhvm cache maybe? (duration: 00m 43s)

Change 344683 had a related patch set uploaded (by Krinkle):
[operations/mediawiki-config@master] StartProfiler: Add hostname in xhgui record

https://gerrit.wikimedia.org/r/344683

Change 344683 merged by jenkins-bot:
[operations/mediawiki-config@master] StartProfiler: Add hostname in xhgui record

https://gerrit.wikimedia.org/r/344683

Mentioned in SAL (#wikimedia-operations) [2017-03-24T19:55:06Z] <krinkle@tin> Synchronized wmf-config/StartProfiler.php: T161286 - include hostname (duration: 00m 49s)

Mentioned in SAL (#wikimedia-operations) [2017-03-24T20:03:49Z] <krinkle@tin> Synchronized php-1.29.0-wmf.17/StartProfiler.php: touch - T161286 - (symlink) (duration: 00m 42s)

Mentioned in SAL (#wikimedia-operations) [2017-03-24T20:03:49Z] <krinkle@tin> Synchronized php-1.29.0-wmf.17/StartProfiler.php: touch - T161286 - (symlink) (duration: 00m 42s)

Ping T126306. File change was effectively ignored by HHVM unless the symlink from $IP/StartProfiler.php to wmf-config/StartProfiler.php` was touched. In addition care must be taken to use touch -h, not touch, as otherwise the destination file is touched..

Krinkle closed this task as Resolved.Mar 24 2017, 8:33 PM
Krinkle claimed this task.
Krinkle triaged this task as Low priority.
Krinkle removed a project: Patch-For-Review.