Page MenuHomePhabricator
Create Task
Maniphest T100694

deployment-videoscaler01 PHP Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fdewiki`.#012 in /srv/mediawiki/multiversion/MWMultiVersion.php on line 369
Closed, ResolvedPublic
Actions

Description

On deployment-videoscaler01 /var/log/syslog is filled with:

May 28 19:05:09 deployment-videoscaler01 php:
   PHP Fatal error:  /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fenwiki/webVideoTranscode%2Fcommonswiki`.#012
  in /srv/mediawiki/multiversion/MWMultiVersion.php on line 369

/var/log/mediawiki/jobrunner.log has a few more info:

2015-05-28T19:05:10+0000: Runner loop 0 process in slot 0 gave status '255':
nice -19 php /srv/mediawiki/multiversion/MWScript.php runJobs.php \
                --wiki='webVideoTranscode%2Fcommonswiki/webVideoTranscode%2Fwikidatawiki' \
                --type='webVideoTranscode%2Fwikidatawiki/webVideoTranscode%2Fdewiki' \
                --maxtime='60' --memory-limit='300M' --result=json
	/srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fcommonswiki/webVideoTranscode%2Fwikidatawiki`.

Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fcommonswiki/webVideoTranscode%2Fwikidatawiki`.
 in /srv/mediawiki/multiversion/MWMultiVersion.php on line 369

The log stated spamming with:

Could not open input file: /srv/deployment/jobrunner/jobrunner/redisJobRunnerService
Could not open input file: /srv/deployment/jobrunner/jobrunner/redisJobRunnerService
Could not open input file: /srv/deployment/jobrunner/jobrunner/redisJobRunnerService
2015-05-27T21:06:53+0000: Starting job spawner loop(s)...
2015-05-27T21:06:53+0000: Initialized loop 0 with 5 runner(s).
2015-05-27T21:06:53+0000: Initialized loop 1 with 1 runner(s).
2015-05-27T21:06:53+0000: Initialized loop 2 with 5 runner(s).
2015-05-27T21:06:53+0000: Initialized loop 3 with 5 runner(s).
2015-05-27T21:06:53+0000: Initialized loop 4 with 1 runner(s).
2015-05-27T21:06:53+0000: Initialized loop 5 with 2 runner(s).
2015-05-27T21:06:55+0000: Runner loop 0 process in slot 0 gave status '255':
nice -19 php /srv/mediawiki/multiversion/MWScript.php runJobs.php --wiki='webVideoTranscode/enwiki' --type='webVideoTranscode%2Fcommonswiki/webVideoTranscode%2Fenwiki' --maxtime='30' --memory-limit='300M' --result=json
        /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode/enwiki`.

Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode/enwiki`.
 in /srv/mediawiki/multiversion/MWMultiVersion.php on line 369

2015-05-27T21:06:55+0000: Runner loop 0 process in slot 3 gave status '255':
nice -19 php /srv/mediawiki/multiversion/MWScript.php runJobs.php --wiki='webVideoTranscode%2Fenwiki/webVideoTranscode%2Fwikidatawiki' --type='1429745518/webVideoTranscode%2Fwikidatawiki' --maxtime='30' --memory-limit='300M' --result=json
        /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fenwiki/webVideoTranscode%2Fwikidatawiki`.

Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fenwiki/webVideoTranscode%2Fwikidatawiki`.
 in /srv/mediawiki/multiversion/MWMultiVersion.php on line 369

2015-05-27T21:06:55+0000: Runner loop 0 process in slot 4 gave status '255':
nice -19 php /srv/mediawiki/multiversion/MWScript.php runJobs.php --wiki='webVideoTranscode%2Fdewiki/webVideoTranscode%2Fenwiki' --type='webVideoTranscode%2Fdewiki/1429745518' --maxtime='30' --memory-limit='300M' --result=json
        /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fenwiki`.

Fatal error: /srv/mediawiki/wikiversions-labs.cdb has no version entry for `webVideoTranscode%2Fdewiki/webVideoTranscode%2Fenwiki`.
 in /srv/mediawiki/multiversion/MWMultiVersion.php on line 369

So something happened on 2015-05-27T21:06:55+0000

Related Objects
Search...

Event Timeline

hashar created this task.May 28 2015, 7:22 PM
hashar raised the priority of this task from to High.
hashar updated the task description. (Show Details)
hashar added projects: Beta-Cluster-Infrastructure, WMF-JobQueue, TimedMediaHandler.
hashar subscribed.
Restricted Application added a project: Multimedia. · View Herald TranscriptMay 28 2015, 7:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
hashar added a comment.May 28 2015, 7:24 PM

Puppet got unbroken roughly at 2015-05-27 14:46:53.029806925 +0000:

Info: Applying configuration version '1432737654'
Notice: /Stage[main]/Mediawiki::Jobrunner/Service[jobchron]/ensure: ensure changed 'stopped' to 'running'

That started the jobchron which run the jobs.

hashar added a subscriber: aaron.May 28 2015, 7:29 PM

@aaron might have an idea.

greg set Security to None.May 28 2015, 7:31 PM
greg added a subscriber: MarkTraceur.
bd808 subscribed.May 28 2015, 8:43 PM

There are strange looking job queues:

127.0.0.1:6379> hkeys jobqueue:aggregator:h-ready-queues:v2
  1) "webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fdewiki/webVideoTranscode%2Fwikidatawiki"
  2) "webVideoTranscode%252Fdewiki%2F1429745518/webVideoTranscode%252Fdewiki%2F1429745518"
  3) "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fcommonswiki/webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fwikidatawiki"
  4) "webVideoTranscode%252Fenwiki%2F1429745518/webVideoTranscode%2Fwikidatawiki"
  5) "webVideoTranscode%2Fwikidatawiki/webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fdewiki"
  6) "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fdewiki/webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fcommonswiki"
  7) "webVideoTranscode%252Fenwiki%2FwebVideoTranscode%252Fenwiki/webVideoTranscode%252Fenwiki%2FwebVideoTranscode%252Fwikidatawiki"
  8) "1429745518%2FwebVideoTranscode%252Fenwiki/webVideoTranscode%252Fenwiki%2FwebVideoTranscode%252Fcommonswiki"
  9) "1429745518%2F1429745518/1429745518"
 10) "webVideoTranscode%2Fdewiki/1429746638"
 11) "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fwikidatawiki/webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fdewiki"
 12) "1429745518%2FwebVideoTranscode%252Fcommonswiki/webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fdewiki"
 13) "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fcommonswiki/webVideoTranscode%2Fenwiki"
 14) "webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fcommonswiki/webVideoTranscode%252Fcommonswiki%2FwebVideoTranscode%252Fcommonswiki"

...

857) "1429745518%2FwebVideoTranscode%252Fwikidatawiki/webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fcommonswiki"
858) "1429745518%2FwebVideoTranscode%252Fdewiki/webVideoTranscode%252Fcommonswiki%2FwebVideoTranscode%252Fwikidatawiki"
859) "webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fdewiki/1429746638"
860) "1429745518%2FwebVideoTranscode%252Fenwiki/webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fcommonswiki"
861) "webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fdewiki/webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fdewiki"
862) "webVideoTranscode%2Fdewiki/webVideoTranscode%252Fcommonswiki%2FwebVideoTranscode%252Fdewiki"
863) "1429745518%2FwebVideoTranscode%252Fcommonswiki/webVideoTranscode%252Fenwiki%2FwebVideoTranscode%252Fcommonswiki"
864) "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fdewiki/EchoNotificationDeleteJob%2Feswiki"
865) "1429745518%2FwebVideoTranscode%252Fcommonswiki/webVideoTranscode%252Fdewiki%2FwebVideoTranscode%252Fenwiki"
866) "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fdewiki/webVideoTranscode%252Fcommonswiki%2FwebVideoTranscode%252Fdewiki"
867) "webVideoTranscode%252Fenwiki%2FwebVideoTranscode%252Fcommonswiki/1429745518%2FwebVideoTranscode%252Fcommonswiki"
868) "webVideoTranscode%252Fwikidatawiki%2FwebVideoTranscode%252Fwikidatawiki/1429745518%2FwebVideoTranscode%252Fdewiki"
hashar added a comment.May 28 2015, 8:46 PM

Some numbers looks like Unix timestamps and are from late April. Maybe some issue occurred at that time and pilled up until they filled all the disk with error?

I would delete them and keep this task open to verify they are no more inserted.

bd808 added a comment.May 28 2015, 8:51 PM

I ran del jobqueue:aggregator:h-ready-queues:v2 on deployment-redis01 to purge this weird job list.

hashar mentioned this in T100806: beta cluster LocalGlobalUserPageCacheUpdateJob attempt to run on db that are not on beta cluster.May 29 2015, 3:40 PM

The log file is filled with a bunch of other error messages. I have filled sub task T100806

hashar claimed this task.Jun 1 2015, 7:16 PM
hashar moved this task from To Triage to In-progress on the Beta-Cluster-Infrastructure board.

Keeping bug open till I am sure this issue is no more happening. Log is still spammed with some other errors though.

hashar moved this task from In-progress to Next: Feature on the Beta-Cluster-Infrastructure board.Jun 10 2015, 8:19 AM
hashar moved this task from Next: Feature to In-progress on the Beta-Cluster-Infrastructure board.
hashar closed this task as Resolved.Jun 10 2015, 1:28 PM
hashar closed subtask T100806: beta cluster LocalGlobalUserPageCacheUpdateJob attempt to run on db that are not on beta cluster as Resolved.

Spam is gone :-}

greg moved this task from In-progress to Done on the Beta-Cluster-Infrastructure board.Jun 10 2015, 4:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL