deployment-pdf01 low free space warning
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• AlexMonk-WMF
	Sep 11 2016, 9:51 PM

Description

WARNING: deployment-prep.deployment-pdf01.diskspace.root.byte_percentfree (<100.00%)
/dev/vda1 7.4G 6.4G 650M 91% /

A non-trivial percentage of that seems to be taken by a single user:

root@deployment-pdf01:/home/cscott# sudo du -hxc | grep -E "(\./[^/]*|total)$" | grep -P "\d*[MG]\t"
469M	./.npm
103M	./tmp
468M	./ocg-collection
1.1G	total

Related Objects

Mentioned In: T131946: puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type"

Event Timeline

• AlexMonk-WMF created this task.Sep 11 2016, 9:51 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 11 2016, 9:51 PM

# du -m --max-depth 2 --one-file-system / |sort -rn|head -n10
6477    /
2738    /usr
2081    /usr/share
1854    /lib
1691    /lib/modules
1039    /home/cscott
1039    /home
547     /srv/deployment
547     /srv
510     /usr/lib

/lib/modules (~ 1.6G) being Linux Kernel compiled modules. Labs has an apt configuration bit that auto upgrade the kernels but never purge them, so that has to be done manually. apt-get autoremove --purge usually get rid of them, have to be careful it does not remove over packages that might have been manually installed.

root@deployment-pdf01:~# apt-get autoremove --purge
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  linux-image-3.13.0-85-generic* linux-image-3.13.0-86-generic*
  linux-image-3.13.0-87-generic* linux-image-3.13.0-88-generic*
  linux-image-3.13.0-91-generic* linux-image-3.13.0-92-generic*
  linux-image-3.13.0-93-generic* linux-image-extra-3.13.0-85-generic*
  linux-image-extra-3.13.0-86-generic* linux-image-extra-3.13.0-87-generic*
  linux-image-extra-3.13.0-88-generic* linux-image-extra-3.13.0-91-generic*
  linux-image-extra-3.13.0-92-generic* linux-image-extra-3.13.0-93-generic*
  linux-tools-3.13.0-85* linux-tools-3.13.0-85-generic* linux-tools-3.13.0-86*
  linux-tools-3.13.0-86-generic* linux-tools-3.13.0-87*
  linux-tools-3.13.0-87-generic* linux-tools-3.13.0-88*
  linux-tools-3.13.0-88-generic* linux-tools-3.13.0-91*
  linux-tools-3.13.0-91-generic* linux-tools-3.13.0-92*
  linux-tools-3.13.0-92-generic* linux-tools-3.13.0-93*
  linux-tools-3.13.0-93-generic*
0 upgraded, 0 newly installed, 28 to remove and 40 not upgraded.
After this operation, 1,378 MB disk space will be freed.
Do you want to continue? [Y/n] 
...

/lib/modules is down from 1.6G to 400M which helps a bit.

@cscott can you look at your ~/.npm on deployment-pdf01 and see whether some material there can be safely deleted? Seems to be mostly npm cache and I don't think it is needed by the PDF rendering service running on that box. Thanks!

hashar triaged this task as Medium priority.Sep 12 2016, 8:38 AM

hashar mentioned this in T131946: puppet failures due to "Could not find class" or "Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type".

greg moved this task from To Triage to Backlog on the Beta-Cluster-Infrastructure board.Sep 20 2016, 5:28 PM

Mentioned in SAL (#wikimedia-releng) [2016-09-26T10:28:46Z] <hashar> beta: on deployment-pdf01 rm -fR /home/cscott/.npm/ T145343

hashar closed this task as Resolved.Sep 26 2016, 10:34 AM

hashar claimed this task.

deployment-pdf01 low free space warningClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

deployment-pdf01 low free space warning
Closed, ResolvedPublic
Actions