There have been a couple of instances in the past weeks (e.g. T116447: [postmortem] Beta Cluster outage: deployment-db2 disk filled up, locked db replication) where runaway queries have taken down Beta Cluster database servers by filling up the /mnt partition which both the datadir and the tmpdir share. We should isolate the latter with its own partition (or quota perhaps) to mitigate the damage in such scenarios. A separate volume for tmpdir is probably the most foolproof setup and should also help performance slighty, though performance is definitely not a major concern.
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
On the labs instances we are using a puppet class that allocates all the extended disk to a single partition /mnt with lvm. End result for db2:
| Filesystem | Size | Used | Avail | Use% | Mounted on |
|---|---|---|---|---|---|
| /dev/mapper/vd-second--local--disk | 147G | 58G | 82G | 42% | /mnt |
The instance has role::mariadb::beta, in theory we could invoke some puppet define to generate two partitions, one for data and the other for tmp.
We would need to resize the existing partition which most probably require MySQL to be shutdown. Hence schedule downtime for beta cluster.