Has had its / filesystem full for some days now, possibly logs?
Description
Details
Project | Branch | Lines +/- | Subject | |
---|---|---|---|---|
operations/mediawiki-config | master | +7 -0 | Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList |
Event Timeline
krenair@deployment-jobrunner02:~$ grep "Aug 10 " /var/log/mediawiki/jobrunner.log -c
7571
krenair@deployment-jobrunner02:~$ grep "Aug 11 " /var/log/mediawiki/jobrunner.log -c
21652810
krenair@deployment-jobrunner02:~$ grep "Aug 12 " /var/log/mediawiki/jobrunner.log -c
107269109
krenair@deployment-jobrunner02:~$ grep "Aug 13 " /var/log/mediawiki/jobrunner.log -c
0
krenair@deployment-jobrunner02:~$ grep "Aug 14 " /var/log/mediawiki/jobrunner.log -c
1078
krenair@deployment-jobrunner02:~$ grep "Aug 15 " /var/log/mediawiki/jobrunner.log -c
0
krenair@deployment-jobrunner02:~$ grep "Aug 16 " /var/log/mediawiki/jobrunner.log -c
0
krenair@deployment-jobrunner02:~$ grep "Aug 17 " /var/log/mediawiki/jobrunner.log -c
0
I moved the file to /srv/jobrunner-old-20170819.log (11GB) and restarted hhvm to prevent apache just responding with 503s (HHVM had failed to load earlier for some reason)
However it's begun to fill up again (I probably should have expected that - it never stopped erroring, it just filled up the drive which prevented it logging more errors earlier)
live-hacked in a fix so wikidata in beta only has valid wikis as clients. also managed to kill the dewikivoyage jobs:
krenair@deployment-tin:/srv/mediawiki-staging$ redis-cli -a $password_here -h deployment-redis01 deployment-redis01:6379> keys dewikivoyage:* 1) "dewikivoyage:jobqueue:ChangeNotification:l-unclaimed" 2) "dewikivoyage:jobqueue:ChangeNotification:h-data" deployment-redis01:6379> del dewikivoyage:jobqueue:ChangeNotification:l-unclaimed (integer) 1 deployment-redis01:6379> del dewikivoyage:jobqueue:ChangeNotification:h-data (integer) 1 deployment-redis01:6379> keys dewikivoyage:* (empty list or set)
Change 372761 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/mediawiki-config@master] Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList
Change 372761 merged by jenkins-bot:
[operations/mediawiki-config@master] Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList
Mentioned in SAL (#wikimedia-operations) [2017-08-21T18:30:01Z] <niharika29@tin> Synchronized wmf-config/Wikibase.php: Wikibase on deployment-prep: Exclude non-existent wikis from clientDbList T173571 (duration: 00m 44s)