gallium and lanthanum disks full (tracking)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Mar 1 2015, 8:08 PM

Description

For a while now (July 2014), disk usage on gallium is going straight down hill. As of January, it's getting a bit more dangerous.

The root disk is 75% full (100GB of 450GB available) and the SSD (used by Jenkins workspaces and gerrit replication) is 75% full (a mere 39GB of 150GB available).

[19:42 UTC] krinkle at gallium.wikimedia.org in ~
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        452G  321G  108G  75% /
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           798M   72M  726M  10% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.9G     0  3.9G   0% /run/shm
/dev/sdb1       149G  111G   39G  75% /srv/ssd
tmpfs           512M     0  512M   0% /var/lib/jenkins-slave/tmpfs

[19:43 UTC] krinkle at gallium.wikimedia.org in /srv/ssd
$ du -sh *
7.6G    gerrit
0       jenkins
96G     jenkins-slave
6.9G    zuul

[19:45 UTC] krinkle at gallium.wikimedia.org in /srv/ssd/jenkins-slave
$ du -sh *
12K     maven..-interceptor.jar
432K    slave.jar
12M     tools
96G     workspace

gallium-disk-year.png (912×2 px, 338 KB)

gallium-mem-year.png (520×806 px, 225 KB)

Related Objects
Search...

Status	Assigned	Task
Resolved	Krinkle	T91211 gallium and lanthanum disks full (tracking)
Resolved	hashar	T91396 Have jenkins jobs logrotate their build history
Declined	hashar	T93189 Write a JJB config tests to ensure all jobs logrotate their build history
Resolved	hashar	T97513 Migrate Jenkins job "operations-puppet-catalog-compiler" to Jenkins Job Builder
Declined	hashar	T93703 reduce copies of mediawiki/core in workspaces
Declined	hashar	T96627 Jenkins jobs must wipe workspace
Declined	hashar	T96629 Convert pool from a few large slaves (4X) to more smaller slaves (1X)
Resolved	Andrew	T96706 Create an instance image like m1.small with 2 CPUs and 30GB space
Resolved	hashar	T97098 Update jobs to use zuul-cloner with git cache
Declined	hashar	T96687 Set up git replication on integration slaves
Resolved	Krinkle	T97106 Zuul-cloner should use hard links when fetching from cache-dir
Resolved	hashar	T118340 Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie
Resolved	Dzahn	T119714 Upload new Zuul packages zuul_2.1.0-60-g1cc37f7-wmf4 on apt.wikimedia.org for Precise / Trusty / Jessie
Resolved	hashar	T119741 integration-zuul-layoutvalidation fails because of python-daemon 2.1
Resolved	hashar	T87294 Nodepool images need Gerrit mirror for git-clone performance
Resolved	Andrew	T113394 Evaluate impact of giant VM images on labs infrastructure
Resolved	hashar	T86659 Migrate all jobs to labs slaves
Resolved	Krinkle	T86175 Migrate jsduck jobs to run in labs (and jsduck-publish via integration-publisher)
Resolved	Krinkle	T86174 Migrate JSDuck to Ubuntu Trusty
Resolved	Krinkle	T93143 Migrate .*testextension-zend jobs to labs slaves
Resolved	hashar	T93559 Migrate mediawiki-core-code-coverage job to labs
Resolved	Krinkle	T93558 Add publishing macro for integration.wikimedia.org/cover
Resolved	hashar	T96463 Move integration-zuul-layout* jobs to labs slaves
Resolved	hashar	T95675 Migrate Jenkins job "Global-Dev Dashboard Data" to JJB and Zuul trigger
Resolved	hashar	T98737 Migrate operations-dns-lint Jenkins job to labs instances
Resolved	hashar	T101966 Remove use of label "hasSlaveScripts" in jobs

Event Timeline

Krinkle created this task.Mar 1 2015, 8:08 PM

Krinkle raised the priority of this task from to High.

Krinkle updated the task description. (Show Details)

Krinkle added projects: Continuous-Integration-Infrastructure, acl*sre-team.

Krinkle subscribed.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 1 2015, 8:08 PM

Krinkle moved this task from Untriaged to Next on the Continuous-Integration-Infrastructure board.Mar 2 2015, 2:47 PM

Running in a screen:

hashar@gallium:/var/lib/jenkins/jobs$ du -sm *|sort -n

Some jobs build history probably need to be logrotated automatically.

hashar claimed this task.Mar 3 2015, 12:02 PM

hashar moved this task from Next to In-progress on the Continuous-Integration-Infrastructure board.

I have canceled the command, instead we can use stat to list the number of entries in each builds directory. The top 30 offenders by number of hardlinks:

cd /var/lib/jenkins/jobs
$ stat --format '%h:%n' */builds|sort -rn|head -n30
13371:operations-puppet-tox-data_admin_lint/builds
12284:mwext-Wikibase-lint/builds
10051:mwext-MobileFrontend-lint/builds
8989:mwext-MobileFrontend-qunit/builds
8983:mwext-MobileFrontend-qunit-mobile/builds
8821:mwext-Flow-lint/builds
8713:operations-apache-config-lint/builds
7512:mwext-VisualEditor-lint/builds
7003:mwext-VisualEditor-npm/builds
7002:mwext-VisualEditor-qunit/builds
6991:mwext-VisualEditor-doc-test/builds
6932:operations-mw-config-tests/builds
5672:mwext-Wikibase-qunit/builds
5656:mwext-Wikibase-repo-tests/builds
5653:mwext-Wikibase-repo-api-tests/builds
5642:mwext-Wikibase-client-tests/builds
5618:mediawiki-core-doxygen-publish/builds
5463:mediawiki-gate/builds
5353:VisualEditor-npm/builds
5096:pywikibot-core-tox-flake8/builds
5040:mediawiki-core-phplint/builds
4863:operations-puppet-validate/builds
4777:VisualEditor-jsduck/builds
4638:mediawiki-core-bundle-rubocop/builds
4422:mwext-Flow-qunit/builds
4384:mwext-MobileFrontend-phpcs-HEAD/builds
4263:mediawiki-core-regression-phpcs-HEAD/builds
3953:pywikibot-core-tox-nose/builds
3920:mwext-MobileFrontend-jslint/builds
3808:operations-puppet-puppetlint-strict/builds

Those jobs are most probably not logrotated.

hashar mentioned this in T91396: Have jenkins jobs logrotate their build history.Mar 3 2015, 2:46 PM

hashar mentioned this in rCICF4bbfbb07e7c4: Logrotate jobs after 30 days by default.Mar 3 2015, 3:45 PM

Krinkle mentioned this in T91997: Replace project-specific "{name}-thing" jobs with generic "thing" ones.Mar 9 2015, 7:53 PM

T91396: Have jenkins jobs logrotate their build history - reduce amount of history kept per build
T91997: Replace project-specific "{name}-thing" jobs with generic "thing" ones - to reduce number of jobs and workspaces left behind in general.
T91410: Automatically delete old jobs not (or no longer) managed by JJB - will remove the build logs of all the jobs we no longer use. Including stuff like the old "regression" and "Nightly - MediaWiki" jobs we used to use. We didn't use logrotate at the time.

Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        452G  223G  206G  52% /

Seems good to me isn't it ?

Resolved for now. Work is in progress to reduce the number of jobs being run that will help keep disk usage at a sane level.

Krinkle moved this task from In-progress to Done on the Continuous-Integration-Infrastructure board.Mar 20 2015, 2:42 AM

Krinkle renamed this task from gallium.wikimedia.org disk space running low to gallium and lanthanum disks full (tracking).Mar 24 2015, 5:50 AM

Krinkle reopened this task as Open.

Krinkle removed hashar as the assignee of this task.

Krinkle removed a project: acl*sre-team.

Krinkle set Security to None.

Krinkle added a subtask: T93703: reduce copies of mediawiki/core in workspaces.

Krinkle added a subtask: T86659: Migrate all jobs to labs slaves.

Krinkle added a subscriber: Legoktm.

Krinkle mentioned this in T93713: All? jobs failing gallium (work fine on lanthanum).Mar 24 2015, 6:25 AM

The on-going efforts on T86659 and T91396 have finally brought gallium and lanthanum to stable levels in terms of disk usage.

hashar closed subtask T91396: Have jenkins jobs logrotate their build history as Resolved.Apr 29 2015, 9:48 AM

hashar closed subtask T93703: reduce copies of mediawiki/core in workspaces as Declined.Feb 19 2016, 8:21 PM

Danny_B added a project: Tracking-Neverending.May 5 2016, 7:38 PM

hashar closed subtask T86659: Migrate all jobs to labs slaves as Resolved.Nov 24 2016, 8:51 PM

Liuxinyu970226 moved this task from Tag to Transition completed / Archived on the Tracking-Neverending board.Mar 21 2019, 10:28 AM

	F108921: lanthanum-mem.png
	Apr 5 2015, 10:01 AM

	F108923: lanthanum-disk.png
	Apr 5 2015, 10:01 AM

	F108922: gallium-disk.png
	Apr 5 2015, 10:01 AM

gallium and lanthanum disks full (tracking)Closed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

gallium and lanthanum disks full (tracking)
Closed, ResolvedPublic
Actions

Related Objects
Search...