contint1001 build cleanup
Closed, ResolvedPublic

Description

Just ran:

1#!/usr/bin/env bash
2
3# remove builds older than 60 days for 25 largest build directories
4while read dir; do
5 sudo find ${dir} -maxdepth 1 -type d -regex '.*/[1-9][0-9]+' -mtime +60 -exec rm -r "{}" \;
6done < <(du -chs /srv/jenkins/builds/* | sort -rh | head -n26 | tail -n25 | awk '{print $2}')
7
8# remove builds older than 30 days for 10 largest build directories
9while read dir; do
10 sudo find ${dir} -maxdepth 1 -type d -regex '.*/[1-9][0-9]+' -mtime +30 -exec rm -r "{}" \;
11done < <(du -chs /srv/jenkins/builds/* | sort -rh | head -n11 | tail -n10 | awk '{print $2}')
12
13# remove builds older than 21 days for 5 largest build directories
14while read dir; do
15 sudo find ${dir} -maxdepth 1 -type d -regex '.*/[1-9][0-9]+' -mtime +21 -exec rm -r "{}" \;
16done < <(du -chs /srv/jenkins/builds/* | sort -rh | head -n6 | tail -n5 | awk '{print $2}')

to clean up old builds on contint1001.

Clean up should be automated and if it is automated then the settings may need to be revisited

Related Objects

Mentioned Here
P7781 clean.sh
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 8 2018, 11:01 PM
thcipriani triaged this task as High priority.Nov 8 2018, 11:02 PM
Legoktm added a subscriber: Legoktm.Nov 9 2018, 2:48 AM

Is this possible to have jenkins do itself?

Jenkins should garbage collect the build artifacts automatically based on jobs configuration. Looking at wmf-quibble-vendor-mysql-php70-docker configuration:

[ X ] Discard old builds
Days to keep builds: 30
Max # of builds to keep: _____
Days to keep artifacts: _____
Max # of builds to keep with artifacts: ____

In JJB there is a global default with:

jjb/defaults.yaml
- defaults:
    name: global
    properties:
     - build-discarder:
         days-to-keep: 30

Which in the XML configuration translate to:

<project>
  <properties>
    <jenkins.model.BuildDiscarderProperty>
      <strategy class="hudson.tasks.LogRotator">
        <daysToKeep>30</daysToKeep>
        <numToKeep>-1</numToKeep>
        <artifactDaysToKeep>-1</artifactDaysToKeep>
        <artifactNumToKeep>-1</artifactNumToKeep>
      </strategy>
    </jenkins.model.BuildDiscarderProperty>
</project>

I guess when I first created those jobs, I have missed overriding that property.

The Quibble jobs are probably generating the most logs, we should probably limit them to 7 days, eventually with an upper limit of XXX builds.

Change 472636 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Only keep 7 days of builds for Quibble jobs

https://gerrit.wikimedia.org/r/472636

There is also the Compress Build Log Plugin. We can probably add support for it in JJB, or just inject the proper XML config to all jobs.

Mentioned in SAL (#wikimedia-operations) [2018-11-12T11:37:09Z] <hashar> contint1001 : cleaning disk | T209123 ?

The worst offenders on contint1001 under /srv/jenkins/builds in MBytes:

5529 ./wikidata-query-rdf-maven
6008 ./phabricator-jessie-commits
6338 ./apps-android-wikipedia-publish
6945 ./mwext-Wikibase-repo-tests-sqlite-hhvm-jessie
8657 ./wikidata-query-rdf
10347 ./analytics-refinery-maven
10489 ./mediawiki-quibble-vendor-sqlite-php70-docker
11349 ./mediawiki-quibble-vendor-mysql-php71-docker
12605 ./quibble-composer-mysql-hhvm-docker
13351 ./mwselenium-quibble-docker
13770 ./wmf-quibble-core-vendor-mysql-php71-docker
14032 ./mediawiki-quibble-vendor-postgres-php70-docker
14250 ./wmf-quibble-core-vendor-mysql-php70-docker
27540 ./quibble-vendor-mysql-php71-docker
27584 ./quibble-vendor-mysql-php70-docker
29208 ./wmf-quibble-vendor-mysql-php71-docker
29223 ./wmf-quibble-vendor-mysql-php70-docker
37536 ./mediawiki-quibble-composer-mysql-php70-docker
40956 ./mediawiki-quibble-vendor-mysql-hhvm-docker
42407 ./wmf-quibble-core-vendor-mysql-hhvm-docker
43002 ./mediawiki-quibble-vendor-mysql-php70-docker
72905 ./quibble-vendor-mysql-hhvm-docker
79850 ./wmf-quibble-vendor-mysql-hhvm-docker
88506 ./apps-android-wikipedia-test

The apps-android-wikipedia-test job generates several apk for the android app, each being roughly 62MBytes:

62	./prod
120	./alpha
63	./dev
62	./fdroid
56	./androidTest
62	./beta
62	./custom
62	./amazon

I am not sure whether we need to capture them.

Change 472636 merged by jenkins-bot:
[integration/config@master] Only keep 7 days of builds for Quibble jobs

https://gerrit.wikimedia.org/r/472636

Mentioned in SAL (#wikimedia-releng) [2018-11-12T12:30:38Z] <hashar> mass purging legacy Jenkins jobs from contint1001. They were still deployed but no more in JJB, that causes the logs / artifacts to be left behind. | T209123

Change 472974 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Only keep 7 days of builds for mediawiki-quibble* jobs

https://gerrit.wikimedia.org/r/472974

I think I got the low hanging fruits solved:

  • Quibble jobs retain the log for 7 days instead of 30 days
  • I have purged a lot of old jobs

Change 472974 merged by jenkins-bot:
[integration/config@master] Only keep 7 days of builds for mediawiki-quibble* jobs

https://gerrit.wikimedia.org/r/472974

greg added a subscriber: greg.

I think I got the low hanging fruits solved:

  • Quibble jobs retain the log for 7 days instead of 30 days
  • I have purged a lot of old jobs

next steps?

hashar closed this task as Resolved.Fri, Nov 30, 3:33 PM
hashar claimed this task.
$ du -d1 -m /srv/jenkins/builds|sort -n|tail -n25
3886	/srv/jenkins/builds/mediawiki-core-code-coverage-docker
4380	/srv/jenkins/builds/mwselenium-quibble-docker
4791	/srv/jenkins/builds/maintenance-disconnect-full-disks
5398	/srv/jenkins/builds/apps-android-wikipedia-publish
6105	/srv/jenkins/builds/wmf-quibble-core-vendor-mysql-php72-docker
6208	/srv/jenkins/builds/wmf-quibble-core-vendor-mysql-php71-docker
6341	/srv/jenkins/builds/wmf-quibble-core-vendor-mysql-php70-docker
7090	/srv/jenkins/builds/mediawiki-quibble-vendor-sqlite-php70-docker
8151	/srv/jenkins/builds/quibble-vendor-mysql-php71-docker
8514	/srv/jenkins/builds/mediawiki-quibble-vendor-mysql-php71-docker
8786	/srv/jenkins/builds/quibble-vendor-mysql-php72-docker
8883	/srv/jenkins/builds/quibble-vendor-mysql-php70-docker
9220	/srv/jenkins/builds/mediawiki-quibble-vendor-mysql-php72-docker
11379	/srv/jenkins/builds/wmf-quibble-vendor-mysql-php72-docker
11472	/srv/jenkins/builds/wmf-quibble-vendor-mysql-php71-docker
11500	/srv/jenkins/builds/wmf-quibble-vendor-mysql-php70-docker
11620	/srv/jenkins/builds/mediawiki-quibble-vendor-postgres-php70-docker
18443	/srv/jenkins/builds/wmf-quibble-core-vendor-mysql-hhvm-docker
24678	/srv/jenkins/builds/quibble-vendor-mysql-hhvm-docker
32922	/srv/jenkins/builds/mediawiki-quibble-vendor-mysql-hhvm-docker
34384	/srv/jenkins/builds/mediawiki-quibble-composer-mysql-php70-docker
34668	/srv/jenkins/builds/mediawiki-quibble-vendor-mysql-php70-docker
35208	/srv/jenkins/builds/wmf-quibble-vendor-mysql-hhvm-docker
94582	/srv/jenkins/builds/apps-android-wikipedia-test

Quibble builds takes twice less disk. The Android ones are still big though. Overall there is less stress put on disk space though.