Change Details

Starting on the 21st the logsize of resourceloader.log grew massively with the uncompressed current log being bigger than 320 GBs and triggering Icinga diskspace alerts for the fluorine log host: -rw-r--r-- 1 udp2log udp2log 29M Jul 18 06:25 resourceloader.log-20160718.gz -rw-r--r-- 1 udp2log udp2log 29M Jul 19 06:24 resourceloader.log-20160719.gz -rw-r--r-- 1 udp2log udp2log 32M Jul 20 06:25 resourceloader.log-20160720.gz -rw-r--r-- 1 udp2log udp2log 887M Jul 21 06:25 resourceloader.log-20160721.gz -rw-r--r-- 1 udp2log udp2log 8.1G Jul 22 06:25 resourceloader.log-20160722.gz -rw-r--r-- 1 udp2log udp2log 14G Jul 23 06:25 resourceloader.log-20160723.gz -rw-r--r-- 1 udp2log udp2log 13G Jul 24 06:25 resourceloader.log-20160724.gz -rw-r--r-- 1 udp2log udp2log 13G Jul 25 06:25 resourceloader.log-20160725.gz -rw-r--r-- 1 udp2log udp2log 15G Jul 26 06:25 resourceloader.log-20160726.gz The log is full of log lines like https://phabricator.wikimedia.org/P3581, e.g. ​2016-07-26 08:58:27 [V5cmMwpAIC0AAFcrsEsAAAAI] mw1175 enwiki 1.28.0-wmf.11 resourceloader DEBUG: Unexpected general module "user" in styles queue. {"module":"user"} https://gerrit.wikimedia.org/r/#/c/301336/ lowered the log level for the resourceloader to "info" as an interim measure.

Starting on the 21st the logsize of resourceloader.log grew massively with the uncompressed current log being bigger than 320 GBs and triggering Icinga diskspace alerts for the fluorine log host: ``` -rw-r--r-- 1 udp2log udp2log 29M Jul 18 06:25 resourceloader.log-20160718.gz -rw-r--r-- 1 udp2log udp2log 29M Jul 19 06:24 resourceloader.log-20160719.gz -rw-r--r-- 1 udp2log udp2log 32M Jul 20 06:25 resourceloader.log-20160720.gz -rw-r--r-- 1 udp2log udp2log 887M Jul 21 06:25 resourceloader.log-20160721.gz -rw-r--r-- 1 udp2log udp2log 8.1G Jul 22 06:25 resourceloader.log-20160722.gz -rw-r--r-- 1 udp2log udp2log 14G Jul 23 06:25 resourceloader.log-20160723.gz -rw-r--r-- 1 udp2log udp2log 13G Jul 24 06:25 resourceloader.log-20160724.gz -rw-r--r-- 1 udp2log udp2log 13G Jul 25 06:25 resourceloader.log-20160725.gz -rw-r--r-- 1 udp2log udp2log 15G Jul 26 06:25 resourceloader.log-20160726.gz ``` The log is full of log lines like P3581 {P3581} https://gerrit.wikimedia.org/r/#/c/301336/ lowered the log level for the resourceloader to "info" as an interim measure.

Starting on the 21st the logsize of resourceloader.log grew massively with the uncompressed current log being bigger than 320 GBs and triggering Icinga diskspace alerts for the fluorine log host: ``` -rw-r--r-- 1 udp2log udp2log 29M Jul 18 06:25 resourceloader.log-20160718.gz -rw-r--r-- 1 udp2log udp2log 29M Jul 19 06:24 resourceloader.log-20160719.gz -rw-r--r-- 1 udp2log udp2log 32M Jul 20 06:25 resourceloader.log-20160720.gz -rw-r--r-- 1 udp2log udp2log 887M Jul 21 06:25 resourceloader.log-20160721.gz -rw-r--r-- 1 udp2log udp2log 8.1G Jul 22 06:25 resourceloader.log-20160722.gz -rw-r--r-- 1 udp2log udp2log 14G Jul 23 06:25 resourceloader.log-20160723.gz -rw-r--r-- 1 udp2log udp2log 13G Jul 24 06:25 resourceloader.log-20160724.gz -rw-r--r-- 1 udp2log udp2log 13G Jul 25 06:25 resourceloader.log-20160725.gz -rw-r--r-- 1 udp2log udp2log 15G Jul 26 06:25 resourceloader.log-20160726.gz ``` The log is full of log lines like P3581 The log is full of log lines like https://phabricator.wikimedia.org/{P3581, e.g.} ​2016-07-26 08:58:27 [V5cmMwpAIC0AAFcrsEsAAAAI] mw1175 enwiki 1.28.0-wmf.11 resourceloader DEBUG: Unexpected general module "user" in styles queue. {"module":"user"} https://gerrit.wikimedia.org/r/#/c/301336/ lowered the log level for the resourceloader to "info" as an interim measure.