Page MenuHomePhabricator

Disk full on deployment-jobrunner02
Closed, ResolvedPublic

Description

Has had its / filesystem full for some days now, possibly logs?

Event Timeline

/var/log/mediawiki/jobrunner.log IIRC
Spam began 11 Aug

krenair@deployment-jobrunner02:~$ grep "Aug 10 " /var/log/mediawiki/jobrunner.log -c
7571
krenair@deployment-jobrunner02:~$ grep "Aug 11 " /var/log/mediawiki/jobrunner.log -c
21652810
krenair@deployment-jobrunner02:~$ grep "Aug 12 " /var/log/mediawiki/jobrunner.log -c
107269109
krenair@deployment-jobrunner02:~$ grep "Aug 13 " /var/log/mediawiki/jobrunner.log -c
0
krenair@deployment-jobrunner02:~$ grep "Aug 14 " /var/log/mediawiki/jobrunner.log -c
1078
krenair@deployment-jobrunner02:~$ grep "Aug 15 " /var/log/mediawiki/jobrunner.log -c
0
krenair@deployment-jobrunner02:~$ grep "Aug 16 " /var/log/mediawiki/jobrunner.log -c
0
krenair@deployment-jobrunner02:~$ grep "Aug 17 " /var/log/mediawiki/jobrunner.log -c
0

I moved the file to /srv/jobrunner-old-20170819.log (11GB) and restarted hhvm to prevent apache just responding with 503s (HHVM had failed to load earlier for some reason)
However it's begun to fill up again (I probably should have expected that - it never stopped erroring, it just filled up the drive which prevented it logging more errors earlier)

live-hacked in a fix so wikidata in beta only has valid wikis as clients. also managed to kill the dewikivoyage jobs:

krenair@deployment-tin:/srv/mediawiki-staging$ redis-cli -a $password_here -h deployment-redis01
deployment-redis01:6379> keys dewikivoyage:*
1) "dewikivoyage:jobqueue:ChangeNotification:l-unclaimed"
2) "dewikivoyage:jobqueue:ChangeNotification:h-data"
deployment-redis01:6379> del dewikivoyage:jobqueue:ChangeNotification:l-unclaimed
(integer) 1
deployment-redis01:6379> del dewikivoyage:jobqueue:ChangeNotification:h-data
(integer) 1
deployment-redis01:6379> keys dewikivoyage:*
(empty list or set)

Change 372761 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/mediawiki-config@master] Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList

https://gerrit.wikimedia.org/r/372761

Change 372761 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList

https://gerrit.wikimedia.org/r/372761

Mentioned in SAL (#wikimedia-operations) [2017-08-21T18:30:01Z] <niharika29@tin> Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)

Krenair claimed this task.

hopefully that should stop this happening again