Special:TimedMediaHandler does not exist and won't even load a webpage
Closed, ResolvedPublic

Description

https://commons.wikimedia.org/wiki/Special:TimedMediaHandler doesn't load Commons at all. Just a white page. No 404, or "Missing page".

Josve05a created this task.Feb 8 2017, 9:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 8 2017, 9:31 PM
Josve05a changed the title from "Special:TimedMediaHandler doe not exist and won't even load a webpage" to "Special:TimedMediaHandler does not exist and won't even load a webpage".Feb 8 2017, 9:31 PM
Josve05a edited the task description. (Show Details)
Revent added a subscriber: Revent.Feb 8 2017, 9:32 PM
Paladox triaged this task as "Unbreak Now!" priority.Feb 8 2017, 9:37 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptFeb 8 2017, 9:37 PM
Paladox added a subscriber: Paladox.Feb 8 2017, 9:38 PM

I Can't reproduce this error locally, i updated my wiki to wmf 11 and it still works i even updated the extension.

https://en.random-wikisaur.tk/wiki/Special:TimedMediaHandler

I Can't reproduce this error locally, i updated my wiki to wmf 11 and it still works i even updated the extension.

https://en.random-wikisaur.tk/wiki/Special:TimedMediaHandler

Try it with 139422 transcodes in the queue. Might be the issue. (Ref)

It is under #Media reports and uploads.

i only see

Media reports and uploads
File list
File usage on other wikis
List of files with duplicates
MIME search
Media statistics
Orphaned TimedText pages
Search for duplicate files
Upload Wizard
Upload Wizard campaigns
Upload file
VIPS scaling test page

i only see

Media reports and uploads
File list
File usage on other wikis
List of files with duplicates
MIME search
Media statistics
Orphaned TimedText pages
Search for duplicate files
Upload Wizard
Upload Wizard campaigns
Upload file
VIPS scaling test page

Unless it is hidden for non-admins, I see it between Search for duplicate files and Upload Wizard.

@mmodell helped me look into this, and he found this

[0e81126f] PHP Fatal Error: request has exceeded memory limit

in the log.

Ok I found the error & stack trace in kibana:

PHP Fatal Error: request has exceeded memory limit

{
  "function": "addByKey",
  "class": "Memcached",
  "type": "::",
  "args": []
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/includes/libs/objectcache/MemcachedBagOStuff.php",
  "line": 79,
  "function": "add",
  "class": "Memcached",
  "object": "[object] (Memcached: {})",
  "type": "->",
  "args": [
    "string",
    "array",
    "integer"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/includes/libs/objectcache/MemcachedPeclBagOStuff.php",
  "line": 173,
  "function": "add",
  "class": "MemcachedBagOStuff",
  "object": "[object] (MemcachedPeclBagOStuff: {})",
  "type": "->",
  "args": [
    "string",
    "array",
    "integer"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/extensions/TimedMediaHandler/SpecialTimedMediaHandler.php",
  "line": 129,
  "function": "add",
  "class": "MemcachedPeclBagOStuff",
  "object": "[object] (MemcachedPeclBagOStuff: {})",
  "type": "->",
  "args": [
    "string",
    "array",
    "integer"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/extensions/TimedMediaHandler/SpecialTimedMediaHandler.php",
  "line": 142,
  "function": "getTranscodes",
  "class": "SpecialTimedMediaHandler",
  "object": "[object] (SpecialTimedMediaHandler: {})",
  "type": "->",
  "args": [
    "string",
    "integer"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/extensions/TimedMediaHandler/SpecialTimedMediaHandler.php",
  "line": 103,
  "function": "getTranscodesTable",
  "class": "SpecialTimedMediaHandler",
  "object": "[object] (SpecialTimedMediaHandler: {})",
  "type": "->",
  "args": [
    "string"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/extensions/TimedMediaHandler/SpecialTimedMediaHandler.php",
  "line": 66,
  "function": "renderState",
  "class": "SpecialTimedMediaHandler",
  "object": "[object] (SpecialTimedMediaHandler: {})",
  "type": "->",
  "args": [
    "OutputPage",
    "string",
    "array",
    "boolean"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/includes/specialpage/SpecialPage.php",
  "line": 522,
  "function": "execute",
  "class": "SpecialTimedMediaHandler",
  "object": "[object] (SpecialTimedMediaHandler: {})",
  "type": "->",
  "args": [
    "NULL"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/includes/specialpage/SpecialPageFactory.php",
  "line": 577,
  "function": "run",
  "class": "SpecialPage",
  "object": "[object] (SpecialTimedMediaHandler: {})",
  "type": "->",
  "args": [
    "NULL"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/includes/MediaWiki.php",
  "line": 285,
  "function": "executePath",
  "class": "SpecialPageFactory",
  "type": "::",
  "args": [
    "Title",
    "RequestContext"
  ]
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/includes/MediaWiki.php",
  "line": 860,
  "function": "performRequest",
  "class": "MediaWiki",
  "object": "[object] (MediaWiki: {})",
  "type": "->",
  "args": []
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/includes/MediaWiki.php",
  "line": 521,
  "function": "main",
  "class": "MediaWiki",
  "object": "[object] (MediaWiki: {})",
  "type": "->",
  "args": []
},
{
  "file": "/srv/mediawiki/php-1.29.0-wmf.11/index.php",
  "line": 43,
  "function": "run",
  "class": "MediaWiki",
  "object": "[object] (MediaWiki: {})",
  "type": "->",
  "args": []
},
{
  "file": "/srv/mediawiki/w/index.php",
  "line": 3,
  "function": "include",
  "args": [
    "string"
  ]
}
demon added a subscriber: demon.

This is from an OOM. Here's the full stacktrace:

(see Mukunda one post above, yay conflict handling)

Is that fixable? Does this mean TMH needs some optimisations regarding the query's it does?

demon added a subscriber: Bawolff.Feb 8 2017, 10:29 PM

So there's no changes between wmf.10 and wmf.11 other than localization updates, so no new suspect code. My guess would be that something's been uploaded today that's causing TMH to choke.

Is that fixable? Does this mean TMH needs some optimisations regarding the query's it does?

I'm sure it is, somehow. Pinging @brion, @Bawolff and @TheDJ as major contributors to this extension.

@Josve05a did this problem happen after upgrading to wmf 11 on commons?

brion added a comment.Feb 8 2017, 10:40 PM

Special:TimedMediaHandler's getTranscodes method asks for every matching row instead of a subset. It's probably just too many files total now. I'll see if I can add a quick hack to limit.

brion added a comment.Feb 8 2017, 10:41 PM

hmm nevermind, should be limiting to 50. Not sure wtf is going on then, looking more closely

Mentioned in SAL (#wikimedia-operations) [2017-02-08T22:43:17Z] <thcipriani> rolling back for wmf.11 from group1 due to T157621

brion added a comment.Feb 8 2017, 10:50 PM

Still fails on .10, I don't think there's any relevant change to the special page in the last version.

Paladox added a comment.EditedFeb 8 2017, 10:50 PM

@thcipriani The problem still happens on wmf 10.

Rolled back group1 from wmf.11 to wmf.10 — page is still not loading. Seems like code changes between branches are likely not to blame in this instance.

@Josve05a did this problem happen after upgrading to wmf 11 on commons?

I'm not sure. I think this happened after @Dispenser started resetting broken transcodes 'en masse', on tens of thousands of files recently.

Mentioned in SAL (#wikimedia-operations) [2017-02-08T23:04:24Z] <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: group1 to 1.29.0-wmf.11 -- T157621 is not code-change related

There are 139k in the queue, and 112k unitialized... that's surely the cause.

brion added a comment.Feb 8 2017, 11:17 PM

It's only loading details for 50 items -- I think the problem is that it's also loading the error output (and then not using it) and some of those are MONSTROUSLY huge, adding up to eating 500 mb of process ram just for the errors:

echo wfGetDB(DB_SLAVE)->selectField('transcode', 'sum(length(transcode_error))', 'transcode_time_startwork IS NOT NULL AND transcode_time_error IS NOT NULL', 'foobar', []);
501118202

Change 336729 had a related patch set uploaded (by Brion VIBBER):
Only load necessary fields on Special:TimedMediaHandler lists

https://gerrit.wikimedia.org/r/336729

@Josve05a did this problem happen after upgrading to wmf 11 on commons?

I'm not sure. I think this happened after @Dispenser started resetting broken transcodes 'en masse', on tens of thousands of files recently.

The problem first occurred on February 6 then disappeared the next day.

Change 336729 merged by jenkins-bot:
Only load necessary fields on Special:TimedMediaHandler lists

https://gerrit.wikimedia.org/r/336729

Change 336734 had a related patch set uploaded (by Paladox):
Only load necessary fields on Special:TimedMediaHandler lists

https://gerrit.wikimedia.org/r/336734

Change 336736 had a related patch set uploaded (by Paladox):
Only load necessary fields on Special:TimedMediaHandler lists

https://gerrit.wikimedia.org/r/336736

TheDJ awarded a token.Feb 9 2017, 7:29 AM
Yann added a subscriber: Yann.Feb 9 2017, 10:38 AM

Change 336734 merged by jenkins-bot:
Only load necessary fields on Special:TimedMediaHandler lists

https://gerrit.wikimedia.org/r/336734

Mentioned in SAL (#wikimedia-operations) [2017-02-09T19:26:49Z] <thcipriani@tin> Synchronized php-1.29.0-wmf.11/extensions/TimedMediaHandler/SpecialTimedMediaHandler.php: SWAT: [[gerrit:336734|Only load necessary fields on Special:TimedMediaHandler lists]] (T157621) (duration: 00m 41s)

brion closed this task as "Resolved".Feb 9 2017, 7:37 PM
brion claimed this task.

Seems happier now; quick fix deployed.

Change 336736 abandoned by Reedy:
Only load necessary fields on Special:TimedMediaHandler lists

https://gerrit.wikimedia.org/r/336736