Page MenuHomePhabricator

Have jenkins jobs logrotate their build history
Closed, ResolvedPublic

Description

Lot of jobs do not discard their build history which is filling up the disk on gallium (T91211). Most of the jobs should have a day based logrotation ie in jjb:

logrotate:
    daysToKeep: 30

Example of top offenders:

cd /var/lib/jenkins/jobs
$ stat --format '%h:%n' */builds|sort -rn|head -n30
13371:operations-puppet-tox-data_admin_lint/builds
12284:mwext-Wikibase-lint/builds
10051:mwext-MobileFrontend-lint/builds
8989:mwext-MobileFrontend-qunit/builds
8983:mwext-MobileFrontend-qunit-mobile/builds
8821:mwext-Flow-lint/builds
8713:operations-apache-config-lint/builds
7512:mwext-VisualEditor-lint/builds
7003:mwext-VisualEditor-npm/builds
7002:mwext-VisualEditor-qunit/builds
6991:mwext-VisualEditor-doc-test/builds
6932:operations-mw-config-tests/builds
5672:mwext-Wikibase-qunit/builds
5656:mwext-Wikibase-repo-tests/builds
5653:mwext-Wikibase-repo-api-tests/builds
5642:mwext-Wikibase-client-tests/builds
5618:mediawiki-core-doxygen-publish/builds
5463:mediawiki-gate/builds
5353:VisualEditor-npm/builds
5096:pywikibot-core-tox-flake8/builds
5040:mediawiki-core-phplint/builds
4863:operations-puppet-validate/builds
4777:VisualEditor-jsduck/builds
4638:mediawiki-core-bundle-rubocop/builds
4422:mwext-Flow-qunit/builds
4384:mwext-MobileFrontend-phpcs-HEAD/builds
4263:mediawiki-core-regression-phpcs-HEAD/builds
3953:pywikibot-core-tox-nose/builds
3920:mwext-MobileFrontend-jslint/builds
3808:operations-puppet-puppetlint-strict/builds

Event Timeline

hashar created this task.Mar 3 2015, 2:46 PM
hashar claimed this task.
hashar raised the priority of this task from to High.
hashar updated the task description. (Show Details)
hashar added subscribers: hashar, Krinkle, Aklapper.

Change 194109 had a related patch set uploaded (by Hashar):
Logrotate tox jobs after 30 days

https://gerrit.wikimedia.org/r/194109

Change 194115 had a related patch set uploaded (by Hashar):
Lower mediawiki-gate logrotation from 90 to 7 days

https://gerrit.wikimedia.org/r/194115

Change 194116 had a related patch set uploaded (by Hashar):
Drop logrotation from 90 days to 30 days

https://gerrit.wikimedia.org/r/194116

Change 194109 merged by jenkins-bot:
Logrotate jobs after 30 days by default

https://gerrit.wikimedia.org/r/194109

Change 194115 merged by jenkins-bot:
Lower mediawiki-gate logrotation from 90 to 7 days

https://gerrit.wikimedia.org/r/194115

hashar updated the task description. (Show Details)Mar 3 2015, 3:51 PM
hashar set Security to None.

Change 194116 merged by jenkins-bot:
Drop logrotation from 90 days to 30 days

https://gerrit.wikimedia.org/r/194116

Still have to check that jobs all have the daysToKeep logrotate parameter.

hashar added a comment.EditedMar 4 2015, 8:53 PM

I have updated most of the jobs. There is still a hundredish that are missing the daysToKeep though. A full list can be gathered via:

ssh gallium.wikimedia.org 'grep -L daysToKeep /var/lib/jenkins/jobs/*/config.xml|cut -d/ -f6'

Ie (as of March 19th):

analytics-libanon
analytics-libcidr                                                                                         
analytics-quarry-web-pep8                                                                                                  
analytics-quarry-web-pyflakes                                                                                                                
analytics-reportcard-data                                                                                                                                
analytics-udp-filters                                                                                                                                        
analytics-webstatscollector                                                                                                                                  
analytics-wikistats                                                                                                                                          
browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce                                                             
cxserver-jslint                                                                                                                                              
Global-Dev Dashboard Data                                                                                                                                    
hashar-mediawiki-core-doxygen-publish
hashar-test-gradle
integration-zuul-config-tox-py27
lucene-search-2-antbuild
mediawiki-core-code-coverage
mediawiki-core-doxygen-publish
mediawiki-core-extensions-integration
mediawiki-core-npm
mediawiki-core-phpunit-hhvm
mediawiki-core-release
mediawiki-language-extension-bundle-integration
mediawiki-phpunit-hhvm
mediawiki-phpunit-zend
mediawiki-selenium-gembuild
mediawiki-vagrant-bundle-rspec
mediawiki-vagrant-bundle-rubocop
mediawiki-vagrant-puppet-doc
mwext-Collection-OfflineContentGenerator-latex_renderer-npm
mwext-Collection-OfflineContentGenerator-npm
mwext-Collection-OfflineContentGenerator-text_renderer-npm
mwext-Collection-OfflineContentGenerator-zim_renderer-npm
mwext-DataValueImplementations-phpcs-HEAD
mwext-DonationInterface-npm
mwext-DonationInterface-runtests
mwext-MobileFrontend-docs
mwext-MobileFrontend-docs-publish
mwext-MultimediaViewer-doc-publish
mwext-MultimediaViewer-doc-test
mwext-MultimediaViewer-do-something
mwext-TemplateData-npm
mwext-UploadWizard-npm
mwext-VisualEditor-csslint
mwext-VisualEditor-doc-publish
mwext-VisualEditor-npm
mwext-VisualEditor-qunit2
mwext-VisualEditor-qunit-karma-debug
mwext-VisualEditor-sync-gerrit
mwext-WikibaseClient-jslint
mwext-WikibaseClient-lint
mwext-WikibaseClient-phpcs-HEAD
mwext-Wikibase-jslint
mwext-Wikibase-phpcs-HEAD
mwext-WikibaseQuery-jslint
mwext-WikibaseQuery-lint
mwext-WikibaseQuery-phpcs-HEAD
mwext-Wikibase-repoapi-tests
mwext-WikibaseRepo-jslint
mwext-WikibaseRepo-lint
mwext-WikibaseRepo-phpcs-HEAD
mwext-Wikidata-client-nonexperimental-tests
mwext-Wikidata-client-tests
mwext-Wikidata-repo-nonexperimental-tests
mwext-Wikidata-repo-tests
mw-tools-codesniffer-composer
mw-tools-releng-tox-doc-rsync
mw-tools-scap-tox-doc-rsync
oojs-ui-composer
oojs-ui-demos-publish
operations-mw-config-composer
operations-puppet-catalog-compiler
operations-puppet-puppetlint
parsoid-php-parsertests
parsoid-regressions
parsoid-roundtrip-test-check
parsoidsvc-deploy-npm
parsoidsvc-deploy-parsertests-run-harder
parsoidsvc-parse-tool-check2
parsoidsvc-parse-tool-check
parsoidsvc-roundtrip-test-check
parsoidsvc-source-npm
parsoidsvc-source-parsertests-run-harder
pywikibot-integration
test-csteipp-sensiolabs-securityadvisorieschecker
test-hashar-java8
test-hashar-myjob
test-krinkle
test-mediawiki-phpcs-HEAD
test-phpmd
translatewiki-puppet-validate
translatewiki-shelllint
utfnormal-composer
visualeditor-doitall2
VisualEditor-qunit
wikidata-gremlin
wikimedia-bugzilla-wikibugs-perllint
wikimedia-fundraising-civicrm-devhand
wikimedia-fundraising-dash-npm
wikimedia-fundraising-DataDashboard-jslint
hashar lowered the priority of this task from High to Normal.Mar 17 2015, 9:54 AM

I have cleaned up a few more jobs. Still have to finish up the whole cleanup though. Maybe we can get a test to ensure all jobs have some logrotate set.

I have cleaned up a few more jobs. Still have to finish up the whole cleanup though. Maybe we can get a test to ensure all jobs have some logrotate set.

Hm.... that seems a bit meta-meta. Adding it to the defaults should be sufficient, right? Any job explicitly removing would need a good reason anyway. Right now I assume the jobs that don't have it yet because 1) the defaults have not been adapted yet, 2) we haven't refreshed the jobs yet.

Either way a test wouldn't help.

You are right, the logrotate in the global defaults and in the other one as well. So a meta test would be overkill probably.

I am keeping this bug around cause I would like to investigate the remaining jobs still missing a logrotate entry. Will update the comment above T91396#1090060 from time to time.

hashar lowered the priority of this task from Normal to Low.Apr 7 2015, 1:44 PM

The bulk of the work has been done though it is not fully completed yet. Keeping this task open as a reminder to complete the work.

Update as of Apr 28th

$ ssh gallium.wikimedia.org 'grep -L daysToKeep /var/lib/jenkins/jobs/*/config.xml|cut -d/ -f6'
analytics-libanon
analytics-libcidr
analytics-reportcard-data
analytics-udp-filters
analytics-webstatscollector
analytics-wikistats
browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce
Global-Dev Dashboard Data
hashar-mediawiki-core-doxygen-publish
hashar-test-gradle
mediawiki-language-extension-bundle-integration
mwext-VisualEditor-sync-gerrit
operations-puppet-catalog-compiler
test-csteipp-sensiolabs-securityadvisorieschecker
test-timed
wikidata-gremlin
wikimedia-fundraising-civicrm-devhand
wikimedia-fundraising-DataDashboard-jslint

Change 207400 had a related patch set uploaded (by Hashar):
Logrotate mwext-VisualEditor-sync-gerrit

https://gerrit.wikimedia.org/r/207400

Change 207401 had a related patch set uploaded (by Hashar):
browsertest for MultimediaViewer win7 ie11

https://gerrit.wikimedia.org/r/207401

Change 207400 merged by jenkins-bot:
Logrotate mwext-VisualEditor-sync-gerrit

https://gerrit.wikimedia.org/r/207400

hashar added a comment.EditedApr 29 2015, 9:30 AM

I have manually changed operations-puppet-catalog-compiler to keep up to 90 days of builds. The job has to be migrated to JJB though which is T97513.

analytics jobs set manually to keep 30 days. JJB migration is T97514

Global-Dev Dashboard Data is T95675

Change 207401 merged by jenkins-bot:
browsertest for MultimediaViewer win7 ie11

https://gerrit.wikimedia.org/r/207401

hashar closed this task as Resolved.Apr 29 2015, 9:48 AM

All jobs now have logrotate. Some needs to be ported to JJB but that is tracked by tasks T97513 T97514 and T95675.

Nothing else left to do :)

The only remaining ones are manual / temporary jobs:

$ ssh gallium.wikimedia.org 'grep -L daysToKeep /var/lib/jenkins/jobs/*/config.xml|cut -d/ -f6'
hashar-mediawiki-core-doxygen-publish
hashar-test-gradle
test-csteipp-sensiolabs-securityadvisorieschecker
wikimedia-fundraising-civicrm-devhand