Incident report for some summary https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWiki
Summary
=======
Some jobs such as **account creation**, **account rename** or **Wikidata dispatcher** invokes `SiteConfiguration::getConfig()` since they have to act on several wikis. That method shells out to `mwscript maintenance/getConfiguration.php` with ulimits being applied, most notably a file size limit of 512MBytes.
However, when HHVM runs the command, it tries to update the byte code cache (either `/var/cache/hhvm/fcgi.sq3` or `/var/cache/hhvm/cli.sq3`). That causes a system error `EFBIG (File too large)` and the job fail.
See [[https://wikitech.wikimedia.org/wiki/Incident_documentation/20160915-MediaWiki incident report]] for actionables.
See below comments for debugging / technical details.
Original task
=========
After pushing 1.28.0-wmf.19 to group1 (which includes wikidatawiki) the dispatch seems to have stopped entirely.
https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch
https://grafana.wikimedia.org/dashboard/db/wikidata-dispatch-script
https://www.wikidata.org/wiki/Special:DispatchStats with wikidata on wmf.19 had:
**Change log statistics**
| ID | Timestamp
| Oldest | 373786394 | 19:45, 12 September 2016
| Newest | 374498192 | 19:48, 15 September 2016
**Dispatch statistics**
| Site | Position | Pending | Lag | Touched
| Freshest | eswikibooks | 374495008 | 3,184 | 30 minutes | 19:18, 15 September 2016
| Median | skwiki | 374494927 | 3,265 | 30 minutes | 19:17, 15 September 2016
| Stalest | fiwiki | 374493639 | 4,553 | 37 minutes | 19:48, 15 September 2016
| Average | - | - | 3,304 | 30 minutes | -
I reverted wikidata back to .18 and the dispatch got handled just fine. Stats view:
Change log statistics
| ID | Timestamp
| Oldest | 373789761 | 20:00, 12 September 2016
| Newest | 374500962 | 20:08, 15 September 2016
Dispatch statistics
| Site | Position | Pending | Lag | Touched
| Freshest | tywiki | 374500941 | 21 | 0 minutes | 20:08, 15 September 2016
| Median | extwiki | 374500920 | 42 | 0 minutes | 20:08, 15 September 2016
| Stalest | bnwikisource | 374500917 | 45 | 0 minutes | 20:08, 15 September 2016
| Average | - | - | 34 | 0 minutes | -
I have noticed:
```
lang=json
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/includes/jobqueue/JobQueueGroup.php",
"line": 422,
"function": "getConfig",
"class": "SiteConfiguration",
"type": "->",
"args": [
"string",
"string"
]
},
{
"function": "{closure}",
"class": "JobQueueGroup",
"type": "->",
"args": [
"boolean",
"integer",
"array",
"NULL"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/includes/libs/objectcache/WANObjectCache.php",
"line": 987,
"function": "call_user_func_array",
"args": [
"Closure",
"array"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/includes/libs/objectcache/WANObjectCache.php",
"line": 892,
"function": "doGetWithSetCallback",
"class": "WANObjectCache",
"type": "->",
"args": [
"string",
"integer",
"Closure",
"array"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/includes/jobqueue/JobQueueGroup.php",
"line": 425,
"function": "getWithSetCallback",
"class": "WANObjectCache",
"type": "->",
"args": [
"string",
"integer",
"Closure",
"array"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/includes/jobqueue/JobQueueGroup.php",
"line": 293,
"function": "getCachedConfigVar",
"class": "JobQueueGroup",
"type": "->",
"args": [
"string"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/includes/jobqueue/JobQueueGroup.php",
"line": 304,
"function": "getQueueTypes",
"class": "JobQueueGroup",
"type": "->",
"args": []
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/includes/jobqueue/JobQueueGroup.php",
"line": 152,
"function": "getDefaultQueueTypes",
"class": "JobQueueGroup",
"type": "->",
"args": []
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/Notifications/JobQueueChangeNotificationSender.php",
"line": 59,
"function": "push",
"class": "JobQueueGroup",
"type": "->",
"args": [
"Wikibase\\ChangeNotificationJob"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/includes/ChangeDispatcher.php",
"line": 248,
"function": "sendNotification",
"class": "Wikibase\\Repo\\Notifications\\JobQueueChangeNotificationSender",
"type": "->",
"args": [
"string",
"array"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/maintenance/dispatchChanges.php",
"line": 210,
"function": "dispatchTo",
"class": "Wikibase\\Repo\\ChangeDispatcher",
"type": "->",
"args": [
"array"
]
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/maintenance/doMaintenance.php",
"line": 110,
"function": "execute",
"class": "Wikibase\\DispatchChanges",
"type": "->",
"args": []
},
{
"file": "/srv/mediawiki/php-1.28.0-wmf.19/extensions/Wikidata/extensions/Wikibase/repo/maintenance/dispatchChanges.php",
"line": 271,
"args": [
"string"
],
"function": "require_once"
},
{
"file": "/srv/mediawiki/multiversion/MWScript.php",
"line": 97,
"args": [
"string"
],
"function": "require_once"
}
```