I have depooled wtp1025 since its root partition was full, causing some mw errors etc..
The rest of the wtp nodes are around 70/75% of usage, the only outlier seems 1025.
The worst dirs on 1025 are:
1.2G /tmp 2.1G /var 4.8G /usr 31G /srv elukey@wtp1025:/srv/mediawiki$ sudo du -hs * | sort -h |tail 340K dblists 2.1M docroot 2.1M portals 8.0M wmf-config 50M static 126M fonts 1.8G php-1.35.0-wmf.24 6.6G php-1.35.0-wmf.40 10G php-1.36.0-wmf.1 11G php-1.35.0-wmf.41 elukey@wtp1025:/srv/mediawiki/php-1.36.0-wmf.1$ sudo du -hs * | sort -h |tail 1.2M HISTORY 4.5M maintenance 18M skins 18M tests 21M resources 30M includes 68M languages 410M vendor 548M extensions 8.9G cache
The reason for all this is that the partition scheme for wtp servers is severely wrong, as it doesn't include a separate SRV partition on a logical volume. We need to reimage all of the WTP servers with a good partition scheme.