Page MenuHomePhabricator
Create Task
Maniphest T130184

beta cluster 'labswiki' not referenced in all-labs.dblist causing jobrunner to error out
Closed, ResolvedPublic
Actions

Description

On beta, the jobrunner attempts to run jobs for labswiki but there is no such database in listed in operations/mediawiki-config:/dblists/all-labs.dblist

The jobrunner RPC call yields a 404:

proxy-server/404 POST http://127.0.0.1:9005/rpc/RunJobs.php?wiki=labswiki&type=htmlCacheUpdate&maxtime=30&maxmem=300M

And:

deployment-tin:~$ mwscript eval.php --wiki=labswiki
no version entry for `labswiki`.

Fatal error: no version entry for `labswiki`.
 in /mnt/srv/mediawiki-staging/multiversion/MWMultiVersion.php on line 365
hashar@deployment-tin:~$

Related Objects
Search...

Event Timeline

hashar created this task.Mar 17 2016, 10:05 AM
hashar added a comment.Mar 17 2016, 10:09 AM

The labswiki on beta cluster has been renamed to deploymentwiki by 352a57e1b18146fa945e0ace91045d2db750dae5 for T72108: Rename labswiki to deploymentwiki.

That is from August 2014 !

hashar added a parent task: T128357: Beta cluster job queue is unmonitored / potentially not running all jobs.Mar 17 2016, 10:29 AM

On deployment-redis01:

> KEYS labswiki*
1) "labswiki:jobqueue:htmlCacheUpdate:h-data"
2) "labswiki:jobqueue:htmlCacheUpdate:h-sha1ById"
3) "labswiki:jobqueue:htmlCacheUpdate:l-unclaimed"
4) "labswiki:jobqueue:htmlCacheUpdate:h-idBySha1"

Each having 2 entries the h-data ones can be grabbed using HGETALL "labswiki:jobqueue:htmlCacheUpdate:h-data"

1) "96f38152075c4ea681a0e8c45c0a5fca"
2) "a:8:{s:4:\"type\";s:15:\"htmlCacheUpdate\";s:9:\"namespace\";i:6;s:5:\"title\";s:11:\"Example.jpg\";s:6:\"params\";a:4:{s:5:\"table\";s:10:\"imagelinks\";s:13:\"rootJobIsSelf\";b:1;s:16:\"rootJobSignature\";s:40:\"5971118571ba6a25041f8b10e47117e33879f48f\";s:16:\"rootJobTimestamp\";s:14:\"20160111192624\";}s:10:\"rtimestamp\";i:0;s:4:\"uuid\";s:32:\"96f38152075c4ea681a0e8c45c0a5fca\";s:4:\"sha1\";s:31:\"3sg5ccb9szl4a90awxfpa589a1j5zbl\";s:9:\"timestamp\";i:1452540384;}"
3) "ac171c5aa1d64381a440011689308ff9"
4) "a:8:{s:4:\"type\";s:15:\"htmlCacheUpdate\";s:9:\"namespace\";i:6;s:5:\"title\";s:48:\"16001000395904-Kiruna-Riksantikvarie\xc3\xa4mbetet.jpg\";s:6:\"params\";a:4:{s:5:\"table\";s:10:\"imagelinks\";s:13:\"rootJobIsSelf\";b:1;s:16:\"rootJobSignature\";s:40:\"1282e28719c03ef4d97fce7d212c2685755eaa92\";s:16:\"rootJobTimestamp\";s:14:\"20160120162710\";}s:10:\"rtimestamp\";i:0;s:4:\"uuid\";s:32:\"ac171c5aa1d64381a440011689308ff9\";s:4:\"sha1\";s:31:\"arulx6tuazyaval49olp1u1iqggk77i\";s:9:\"timestamp\";i:1453307230;}"

They are from January 2016. No idea how they went enqueued since labswiki is no more. Did the deletes:

> HDEL "labswiki:jobqueue:htmlCacheUpdate:h-data" "96f38152075c4ea681a0e8c45c0a5fca"
> HDEL "labswiki:jobqueue:htmlCacheUpdate:h-data" "ac171c5aa1d64381a440011689308ff9"
> HGETALL "labswiki:jobqueue:htmlCacheUpdate:h-data"
(empty list or set)

I then deleted the labswiki keys:

> KEYS labswiki*
1) "labswiki:jobqueue:htmlCacheUpdate:h-sha1ById"
2) "labswiki:jobqueue:htmlCacheUpdate:l-unclaimed"
3) "labswiki:jobqueue:htmlCacheUpdate:h-idBySha1"
> DEL "labswiki:jobqueue:htmlCacheUpdate:h-sha1ById"
(integer) 1
> DEL "labswiki:jobqueue:htmlCacheUpdate:h-idBySha1"
(integer) 1
> DEL "labswiki:jobqueue:htmlCacheUpdate:l-unclaimed"
(integer) 1
> KEYS labswiki*
(empty list or set)

404 are gone from jobrunner01 /var/log/apache2/jobqueue-access.log

hashar closed this task as Resolved.Mar 17 2016, 10:29 AM
hashar claimed this task.
hashar mentioned this in T128357: Beta cluster job queue is unmonitored / potentially not running all jobs.
hashar mentioned this in T130179: deployment-jobrunner01/Free space - all mounts is CRITICAL.Mar 17 2016, 10:32 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL