ATM it might happen that on a swift backend host the root filesystem gets filled up:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| swift: change ownership depending on mountpoint status | operations/puppet | production | +22 -6 |
Change 516615 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] swift: change ownership depending on mountpoint status
One thing that can happen with this changeset, in the case where a disk bounces like that repeatedly, is that before each puppet run chowns the directory to root, some additional data is copied to the local filesystem. After a few cycles that can fill / and we'll be back here again.
Can we get alerts on flapping disks rather, and keep them from being remounted if they have errors? Or can we do that in addition to the proposed patch?
The remounting is something we need to investigate why for sure, although related I think there are two independent issues. To be clear the intent of the patch above is to address a disk being e.g. waiting to be replaced and making sure the correct permissions are in place. IOW we'd have this issue even if remounting (by systemd I believe) wasn't happening
Change 516615 merged by Filippo Giunchedi:
[operations/puppet@production] swift: change ownership depending on mountpoint status
This is fixed, permissions will be set depending on whether the directory is a mountpoint or not. Unrelated but distinct issue is the interaction between drive-audit and systemd in T265450: flip/flop mounting filesystems between systemd and swift-drive-audit