Page MenuHomePhabricator
Create Task
Maniphest T91396

Have jenkins jobs logrotate their build history
Closed, ResolvedPublic
Actions

Description

Lot of jobs do not discard their build history which is filling up the disk on gallium (T91211). Most of the jobs should have a day based logrotation ie in jjb:

logrotate:
    daysToKeep: 30

Example of top offenders:

cd /var/lib/jenkins/jobs
$ stat --format '%h:%n' */builds|sort -rn|head -n30
13371:operations-puppet-tox-data_admin_lint/builds
12284:mwext-Wikibase-lint/builds
10051:mwext-MobileFrontend-lint/builds
8989:mwext-MobileFrontend-qunit/builds
8983:mwext-MobileFrontend-qunit-mobile/builds
8821:mwext-Flow-lint/builds
8713:operations-apache-config-lint/builds
7512:mwext-VisualEditor-lint/builds
7003:mwext-VisualEditor-npm/builds
7002:mwext-VisualEditor-qunit/builds
6991:mwext-VisualEditor-doc-test/builds
6932:operations-mw-config-tests/builds
5672:mwext-Wikibase-qunit/builds
5656:mwext-Wikibase-repo-tests/builds
5653:mwext-Wikibase-repo-api-tests/builds
5642:mwext-Wikibase-client-tests/builds
5618:mediawiki-core-doxygen-publish/builds
5463:mediawiki-gate/builds
5353:VisualEditor-npm/builds
5096:pywikibot-core-tox-flake8/builds
5040:mediawiki-core-phplint/builds
4863:operations-puppet-validate/builds
4777:VisualEditor-jsduck/builds
4638:mediawiki-core-bundle-rubocop/builds
4422:mwext-Flow-qunit/builds
4384:mwext-MobileFrontend-phpcs-HEAD/builds
4263:mediawiki-core-regression-phpcs-HEAD/builds
3953:pywikibot-core-tox-nose/builds
3920:mwext-MobileFrontend-jslint/builds
3808:operations-puppet-puppetlint-strict/builds

Details

SubjectRepoBranchLines +/-
browsertest for MultimediaViewer win7+ie11integration/configmaster+5 -0
Logrotate mwext-VisualEditor-sync-gerritintegration/configmaster+2 -0
Drop logrotation from 90 days to 30 daysintegration/configmaster+0 -45
Lower mediawiki-gate logrotation from 90 to 7 daysintegration/configmaster+1 -1
Logrotate jobs after 30 days by defaultintegration/configmaster+18 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Mar 3 2015, 2:46 PM
hashar claimed this task.
hashar raised the priority of this task from to High.
hashar updated the task description. (Show Details)
hashar added a project: Continuous-Integration-Infrastructure.
hashar added subscribers: hashar, Krinkle, Aklapper.
gerritbot subscribed.Mar 3 2015, 2:47 PM

Change 194109 had a related patch set uploaded (by Hashar):
Logrotate tox jobs after 30 days

https://gerrit.wikimedia.org/r/194109

gerritbot added a project: Patch-For-Review.Mar 3 2015, 2:47 PM
gerritbot added a comment.Mar 3 2015, 3:16 PM

Change 194115 had a related patch set uploaded (by Hashar):
Lower mediawiki-gate logrotation from 90 to 7 days

https://gerrit.wikimedia.org/r/194115

gerritbot added a comment.Mar 3 2015, 3:18 PM

Change 194116 had a related patch set uploaded (by Hashar):
Drop logrotation from 90 days to 30 days

https://gerrit.wikimedia.org/r/194116

hashar moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.Mar 3 2015, 3:20 PM
gerritbot added a comment.Mar 3 2015, 3:45 PM

Change 194109 merged by jenkins-bot:
Logrotate jobs after 30 days by default

https://gerrit.wikimedia.org/r/194109

gerritbot added a comment.Mar 3 2015, 3:45 PM

Change 194115 merged by jenkins-bot:
Lower mediawiki-gate logrotation from 90 to 7 days

https://gerrit.wikimedia.org/r/194115

hashar mentioned this in rCICF4bbfbb07e7c4: Logrotate jobs after 30 days by default.Mar 3 2015, 3:45 PM
hashar mentioned this in rCICF8ed4991c026f: Lower mediawiki-gate logrotation from 90 to 7 days.
hashar updated the task description. (Show Details)Mar 3 2015, 3:51 PM
hashar set Security to None.
gerritbot added a comment.Mar 4 2015, 10:38 AM

Change 194116 merged by jenkins-bot:
Drop logrotation from 90 days to 30 days

https://gerrit.wikimedia.org/r/194116

hashar mentioned this in rCICF91a689084974: Drop logrotation from 90 days to 30 days.Mar 4 2015, 10:38 AM
hashar added a comment.Mar 4 2015, 10:38 AM

Still have to check that jobs all have the daysToKeep logrotate parameter.

hashar added a comment.EditedMar 4 2015, 8:53 PM

I have updated most of the jobs. There is still a hundredish that are missing the daysToKeep though. A full list can be gathered via:

ssh gallium.wikimedia.org 'grep -L daysToKeep /var/lib/jenkins/jobs/*/config.xml|cut -d/ -f6'

Ie (as of March 19th):

analytics-libanon
analytics-libcidr                                                                                         
analytics-quarry-web-pep8                                                                                                  
analytics-quarry-web-pyflakes                                                                                                                
analytics-reportcard-data                                                                                                                                
analytics-udp-filters                                                                                                                                        
analytics-webstatscollector                                                                                                                                  
analytics-wikistats                                                                                                                                          
browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce                                                             
cxserver-jslint                                                                                                                                              
Global-Dev Dashboard Data                                                                                                                                    
hashar-mediawiki-core-doxygen-publish
hashar-test-gradle
integration-zuul-config-tox-py27
lucene-search-2-antbuild
mediawiki-core-code-coverage
mediawiki-core-doxygen-publish
mediawiki-core-extensions-integration
mediawiki-core-npm
mediawiki-core-phpunit-hhvm
mediawiki-core-release
mediawiki-language-extension-bundle-integration
mediawiki-phpunit-hhvm
mediawiki-phpunit-zend
mediawiki-selenium-gembuild
mediawiki-vagrant-bundle-rspec
mediawiki-vagrant-bundle-rubocop
mediawiki-vagrant-puppet-doc
mwext-Collection-OfflineContentGenerator-latex_renderer-npm
mwext-Collection-OfflineContentGenerator-npm
mwext-Collection-OfflineContentGenerator-text_renderer-npm
mwext-Collection-OfflineContentGenerator-zim_renderer-npm
mwext-DataValueImplementations-phpcs-HEAD
mwext-DonationInterface-npm
mwext-DonationInterface-runtests
mwext-MobileFrontend-docs
mwext-MobileFrontend-docs-publish
mwext-MultimediaViewer-doc-publish
mwext-MultimediaViewer-doc-test
mwext-MultimediaViewer-do-something
mwext-TemplateData-npm
mwext-UploadWizard-npm
mwext-VisualEditor-csslint
mwext-VisualEditor-doc-publish
mwext-VisualEditor-npm
mwext-VisualEditor-qunit2
mwext-VisualEditor-qunit-karma-debug
mwext-VisualEditor-sync-gerrit
mwext-WikibaseClient-jslint
mwext-WikibaseClient-lint
mwext-WikibaseClient-phpcs-HEAD
mwext-Wikibase-jslint
mwext-Wikibase-phpcs-HEAD
mwext-WikibaseQuery-jslint
mwext-WikibaseQuery-lint
mwext-WikibaseQuery-phpcs-HEAD
mwext-Wikibase-repoapi-tests
mwext-WikibaseRepo-jslint
mwext-WikibaseRepo-lint
mwext-WikibaseRepo-phpcs-HEAD
mwext-Wikidata-client-nonexperimental-tests
mwext-Wikidata-client-tests
mwext-Wikidata-repo-nonexperimental-tests
mwext-Wikidata-repo-tests
mw-tools-codesniffer-composer
mw-tools-releng-tox-doc-rsync
mw-tools-scap-tox-doc-rsync
oojs-ui-composer
oojs-ui-demos-publish
operations-mw-config-composer
operations-puppet-catalog-compiler
operations-puppet-puppetlint
parsoid-php-parsertests
parsoid-regressions
parsoid-roundtrip-test-check
parsoidsvc-deploy-npm
parsoidsvc-deploy-parsertests-run-harder
parsoidsvc-parse-tool-check2
parsoidsvc-parse-tool-check
parsoidsvc-roundtrip-test-check
parsoidsvc-source-npm
parsoidsvc-source-parsertests-run-harder
pywikibot-integration
test-csteipp-sensiolabs-securityadvisorieschecker
test-hashar-java8
test-hashar-myjob
test-krinkle
test-mediawiki-phpcs-HEAD
test-phpmd
translatewiki-puppet-validate
translatewiki-shelllint
utfnormal-composer
visualeditor-doitall2
VisualEditor-qunit
wikidata-gremlin
wikimedia-bugzilla-wikibugs-perllint
wikimedia-fundraising-civicrm-devhand
wikimedia-fundraising-dash-npm
wikimedia-fundraising-DataDashboard-jslint
Krinkle mentioned this in T91211: gallium and lanthanum disks full (tracking).Mar 9 2015, 7:57 PM
hashar lowered the priority of this task from High to Medium.Mar 17 2015, 9:54 AM

I have cleaned up a few more jobs. Still have to finish up the whole cleanup though. Maybe we can get a test to ensure all jobs have some logrotate set.

hashar mentioned this in T93189: Write a JJB config tests to ensure all jobs logrotate their build history.Mar 19 2015, 11:00 AM
Krinkle added a comment.Mar 21 2015, 6:35 AM
In T91396#1124540, @hashar wrote:

I have cleaned up a few more jobs. Still have to finish up the whole cleanup though. Maybe we can get a test to ensure all jobs have some logrotate set.

Hm.... that seems a bit meta-meta. Adding it to the defaults should be sufficient, right? Any job explicitly removing would need a good reason anyway. Right now I assume the jobs that don't have it yet because 1) the defaults have not been adapted yet, 2) we haven't refreshed the jobs yet.

Either way a test wouldn't help.

hashar mentioned this in rCICF025febc17ae3: Logrotate search* jobs after 15 days.Mar 23 2015, 3:56 PM
hashar mentioned this in rCICFea42e9e88eb7: Set build log rotation on some beta jobs.
Krinkle moved this task from In-progress to Done on the Continuous-Integration-Infrastructure board.Mar 25 2015, 3:33 AM
Krinkle moved this task from Done to In-progress on the Continuous-Integration-Infrastructure board.
hashar closed subtask T93189: Write a JJB config tests to ensure all jobs logrotate their build history as Declined.Mar 26 2015, 11:39 AM
hashar added a comment.Mar 26 2015, 11:41 AM

You are right, the logrotate in the global defaults and in the other one as well. So a meta test would be overkill probably.

I am keeping this bug around cause I would like to investigate the remaining jobs still missing a logrotate entry. Will update the comment above T91396#1090060 from time to time.

hashar lowered the priority of this task from Medium to Low.Apr 7 2015, 1:44 PM

The bulk of the work has been done though it is not fully completed yet. Keeping this task open as a reminder to complete the work.

hashar added a comment.Apr 28 2015, 2:15 PM

Update as of Apr 28th

$ ssh gallium.wikimedia.org 'grep -L daysToKeep /var/lib/jenkins/jobs/*/config.xml|cut -d/ -f6'
analytics-libanon
analytics-libcidr
analytics-reportcard-data
analytics-udp-filters
analytics-webstatscollector
analytics-wikistats
browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce
Global-Dev Dashboard Data
hashar-mediawiki-core-doxygen-publish
hashar-test-gradle
mediawiki-language-extension-bundle-integration
mwext-VisualEditor-sync-gerrit
operations-puppet-catalog-compiler
test-csteipp-sensiolabs-securityadvisorieschecker
test-timed
wikidata-gremlin
wikimedia-fundraising-civicrm-devhand
wikimedia-fundraising-DataDashboard-jslint
gerritbot added a comment.Apr 29 2015, 9:22 AM

Change 207400 had a related patch set uploaded (by Hashar):
Logrotate mwext-VisualEditor-sync-gerrit

https://gerrit.wikimedia.org/r/207400

gerritbot added a comment.Apr 29 2015, 9:22 AM

Change 207401 had a related patch set uploaded (by Hashar):
browsertest for MultimediaViewer win7 ie11

https://gerrit.wikimedia.org/r/207401

gerritbot added a comment.Apr 29 2015, 9:24 AM

Change 207400 merged by jenkins-bot:
Logrotate mwext-VisualEditor-sync-gerrit

https://gerrit.wikimedia.org/r/207400

hashar mentioned this in rCICF117c6f211a31: Logrotate mwext-VisualEditor-sync-gerrit.Apr 29 2015, 9:25 AM
hashar added a subtask: T97513: Migrate Jenkins job "operations-puppet-catalog-compiler" to Jenkins Job Builder .Apr 29 2015, 9:27 AM
hashar added a comment.EditedApr 29 2015, 9:30 AM

I have manually changed operations-puppet-catalog-compiler to keep up to 90 days of builds. The job has to be migrated to JJB though which is T97513.

analytics jobs set manually to keep 30 days. JJB migration is T97514

Global-Dev Dashboard Data is T95675

gerritbot added a comment.Apr 29 2015, 9:31 AM

Change 207401 merged by jenkins-bot:
browsertest for MultimediaViewer win7 ie11

https://gerrit.wikimedia.org/r/207401

hashar mentioned this in rCICF31056399a81c: browsertest for MultimediaViewer win7+ie11.Apr 29 2015, 9:31 AM
hashar closed this task as Resolved.Apr 29 2015, 9:48 AM

All jobs now have logrotate. Some needs to be ported to JJB but that is tracked by tasks T97513 T97514 and T95675.

Nothing else left to do :)

hashar added a comment.Apr 29 2015, 9:49 AM

The only remaining ones are manual / temporary jobs:

$ ssh gallium.wikimedia.org 'grep -L daysToKeep /var/lib/jenkins/jobs/*/config.xml|cut -d/ -f6'
hashar-mediawiki-core-doxygen-publish
hashar-test-gradle
test-csteipp-sensiolabs-securityadvisorieschecker
wikimedia-fundraising-civicrm-devhand
hashar moved this task from In-progress to Done on the Continuous-Integration-Infrastructure board.Apr 29 2015, 9:49 AM
hashar closed subtask T97513: Migrate Jenkins job "operations-puppet-catalog-compiler" to Jenkins Job Builder as Resolved.May 5 2020, 1:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits