graphite.wmflabs.org no longer purges data for deleted instances
Open, LowPublic
Actions

Assigned To

None

Authored By

	Krinkle
	Mar 25 2015, 9:35 AM

Description

https://tools.wmflabs.org/nagf/?project=integration is becoming more and more confusing due to deleted instances being retained. This in combination with Diamond's weird echoing behaviour (where the last data point is echoed and repeated; even if the node no longer exists) makes things hard to monitor.

I recall this being a problem last year, but from what I remember this was fixed by @yuvipanda. Looks like it might have regressed.

The Graphite index includes deleted instances. This is solved by querying wikitech to discover instance names.
Wildcard query (e.g. integration.* for cluster overview) includes metrics from deleted instances. For example, https://tools.wmflabs.org/nagf/?project=deployment-prep "cluster Puppet agent" shows 2 puppet failures even though none of the listed instances have puppet failures. This is because Diamond keeps echoing puppet failures from instances that no longer exist.
The Graphite index includes deleted metrics (disk_space.* shows mounts that no longer exist). E.g. a dangerously low /var on an instance that actually no longer has that mount.

Related Objects
Search...

Status	Assigned	Task
Open	None	T93861 graphite.wmflabs.org no longer purges data for deleted instances
Resolved	• Phamhi	T218185 labmon1001 running low on disk on /srv
Resolved	taavi	T218192 fullstackd - Do not collect metrics about the VMs themselves, only the test process
Invalid	None	T120377 labmon1001 graphite instance archiver keeps archiving the same instances

Event Timeline

Krinkle created this task.Mar 25 2015, 9:35 AM

Krinkle raised the priority of this task from to Needs Triage.

Krinkle updated the task description. (Show Details)

Krinkle added projects: Cloud-Services, Cloud-VPS.

Krinkle added subscribers: Krinkle, yuvipanda.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 25 2015, 9:35 AM

Krinkle updated the task description. (Show Details)Mar 25 2015, 9:36 AM

Krinkle set Security to None.

Yeah, I reverted my fix because there were bugs in that script and it ended up filling the disk with 'archived' data. See Id5f026abe1ac2de99d962ea9a8777598cb304e15

Also you should use the wikitech API (wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niproject=deployment-prep&niregion=eqiad&format=json) to get list of current instances than rely on graphite.

This is not limited to entire instances. It also applies to individual data points. For example, this graph for disk space on integration-puppetmaster shows a steady ~ 700MB free on /var when in reality it got unmounted (or rather, we recreated the instance without that mount).

graphite.wmflabs.png (250×400 px, 13 KB)

In T93861#1148096, @yuvipanda wrote:

Also you should use the wikitech API (wikitech.wikimedia.org/w/api.php?action=query&list=novainstances&niproject=deployment-prep&niregion=eqiad&format=json) to get list of current instances than rely on graphite.

Cool. I'll do that. Filed https://github.com/wikimedia/nagf/issues/2.

The issue with data points remains I suppose.

(Needsvolunteer :)). Also replacing txstatsd might help

Krinkle mentioned this in T94514: novainstances API: 'niregion' parameter should be optional and/or discoverable.Mar 31 2015, 12:08 AM

Krinkle updated the task description. (Show Details)Mar 31 2015, 4:22 PM

Krinkle mentioned this in T95569: Purge graphite data for deleted integration instances and nonexistent metrics.Apr 9 2015, 4:08 PM

yuvipanda mentioned this in T99077: Shinken: make sure 'Free space - all mounts' can handle no-longer-existing mounts.May 14 2015, 8:32 PM

Krenair subscribed.Oct 13 2015, 1:30 AM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:53 PM

• GTirloni added a subtask: T218185: labmon1001 running low on disk on /srv.Mar 24 2019, 10:25 AM

• GTirloni edited projects, added cloud-services-team (Kanban); removed Cloud-Services.

• GTirloni added a subscriber: yuvipanda.