The dumps distribution servers, labstore1006.wikimedia.org and labstore1007.wikimedia.org, are filling up rather quickly. They are also out of warranty and have both acquired issues that cause random crashes (see T268280: labstore1006 spontaneous reboot and now T281045: labstore1007 crashed after storage controller errors--replace disk?).
It is the nature of the service to increase usage over time, but storage usage appears to have increased a bit faster recently. Also, the storage usage is surprisingly uneven.
- We should take any actions we can to try to reduce space consumption while we figure out expansion
- Also, we need to decide what "expansion" looks like since these servers are already problematic, but they lack a support contract
- Currently both are in racks with no room whatsoever. Shelves for them are 2U, and each can support 2 more shelves (doubling current space). However, there's nowhere to put them.