fullstackd runs every 5 minutes to test VM creation (including Puppet tests).
This generates a huge amount of data about VMs we don't care about (~800GB for 3 months).
We should verify whether our monitoring depends on this data (hopefully not) and disable diamond in VMs created by fullstackd.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T93861 graphite.wmflabs.org no longer purges data for deleted instances | |||
| Resolved | • Phamhi | T218185 labmon1001 running low on disk on /srv | |||
| Resolved | taavi | T218192 fullstackd - Do not collect metrics about the VMs themselves, only the test process |
With T218365 fixed, we could add diamond::remove to project-admin's Hiera. The only instances that exist that are transient (or failures) created by fullstackd.
When this hiera value is applied, the initial Puppet run shouldn't install diamond (::standard::diamond.pp, line 22).
The metrics coming out of fullstackd are in Graphite under the servers.cloudcontrol1003.nova tree.
fullstackd instances are now on Bullseye which does not included Diamond, and metricsinfra prometheus has been configured to ignore these instances.