Page MenuHomePhabricator

labmon1001 graphite instance archiver keeps archiving the same instances
Closed, InvalidPublic

Description

noticed while following up on instance-archiver, looks like the same three get archived over and over again

root@labmon1001:~# grep -e Found -e Archived /var/log/graphite/instance-archiver.log | tail -30
2015-12-04 08:04:08,568 Archived host eqiad, renamed to 20151204080408-eqiad
2015-12-04 08:04:08,568 Archived host rate, renamed to 20151204080408-rate
2015-12-04 09:04:06,810 Found 3 host(s) in 3 project(s) to archive
2015-12-04 09:04:07,315 Archived host wikimetrics, renamed to 20151204090406-wikimetrics
2015-12-04 09:04:07,315 Archived host eqiad, renamed to 20151204090407-eqiad
2015-12-04 09:04:07,315 Archived host rate, renamed to 20151204090407-rate
2015-12-04 10:03:54,120 Found 3 host(s) in 3 project(s) to archive
2015-12-04 10:03:58,351 Archived host eqiad, renamed to 20151204100354-eqiad
2015-12-04 10:03:58,352 Archived host rate, renamed to 20151204100358-rate
2015-12-04 10:03:58,352 Archived host wikimetrics, renamed to 20151204100358-wikimetrics
2015-12-04 11:04:01,413 Found 3 host(s) in 3 project(s) to archive
2015-12-04 11:04:06,670 Archived host rate, renamed to 20151204110401-rate
2015-12-04 11:04:06,671 Archived host eqiad, renamed to 20151204110406-eqiad
2015-12-04 11:04:06,672 Archived host wikimetrics, renamed to 20151204110406-wikimetrics
2015-12-04 12:03:50,833 Found 3 host(s) in 3 project(s) to archive
2015-12-04 12:03:55,157 Archived host eqiad, renamed to 20151204120350-eqiad
2015-12-04 12:03:55,158 Archived host wikimetrics, renamed to 20151204120355-wikimetrics
2015-12-04 12:03:55,158 Archived host rate, renamed to 20151204120355-rate
2015-12-04 13:04:09,179 Found 3 host(s) in 3 project(s) to archive
2015-12-04 13:04:10,393 Archived host rate, renamed to 20151204130409-rate
2015-12-04 13:04:10,393 Archived host wikimetrics, renamed to 20151204130410-wikimetrics
2015-12-04 13:04:10,393 Archived host eqiad, renamed to 20151204130410-eqiad
2015-12-04 14:03:57,004 Found 3 host(s) in 3 project(s) to archive
2015-12-04 14:04:01,455 Archived host eqiad, renamed to 20151204140356-eqiad
2015-12-04 14:04:01,456 Archived host wikimetrics, renamed to 20151204140401-wikimetrics
2015-12-04 14:04:01,456 Archived host rate, renamed to 20151204140401-rate
2015-12-04 15:03:55,882 Found 3 host(s) in 3 project(s) to archive
2015-12-04 15:03:59,400 Archived host eqiad, renamed to 20151204150355-eqiad
2015-12-04 15:03:59,401 Archived host rate, renamed to 20151204150359-rate
2015-12-04 15:03:59,401 Archived host wikimetrics, renamed to 20151204150359-wikimetrics

Event Timeline

fgiunchedi raised the priority of this task from to Medium.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added projects: SRE, Grafana.
fgiunchedi added subscribers: fgiunchedi, yuvipanda.

I think the cause is metrics that are flowing into labmon1001 but don't really belong to a project, their prefix happens to have the same name as a project, namely varnish.eqiad, logstash.rate, analytics.wikimetrics,

Krinkle renamed this task from graphite instance archiver keeps archiving the same instances to labmon1001 graphite instance archiver keeps archiving the same instances.Dec 4 2015, 4:35 PM
Krinkle set Security to None.

Change 258976 had a related patch set uploaded (by Filippo Giunchedi):
graphite: run archive-instance once a day

https://gerrit.wikimedia.org/r/258976

Change 258976 merged by Filippo Giunchedi:
graphite: run archive-instance once a day

https://gerrit.wikimedia.org/r/258976

Bstorm subscribed.

We can't even tell what this relates to anymore.