Data collection for tools job_count seems to be broken
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bd808
	Oct 31 2016, 9:56 PM

Description

https://graphite-labs.wikimedia.org/render/?width=600&height=300&target=cactiStyle(sumSeries(tools.tools-services-01.sge.hosts.tools*.job_count))&from=-1d

graphite-labs.wikimedia.org.png (300×600 px, 21 KB)

Sometime around 2016-10-31T15:00 the collected job counts for the tools.tools-services-01.sge.hosts.tools*.job_count graphite metrics dropped from ~800 to ~150.

Some early investigation seems to point to a failure in the collector related to the removal of the defunct tools-exec-cyberbot job runner.

Related Objects

Mentioned In: T154539: Reduce Precise OGE exec hosts to 5
T151980: Reduce Precise OGE exec hosts to 10

Event Timeline

bd808 created this task.Oct 31 2016, 9:56 PM

Restricted Application added a project: Cloud-Services. · View Herald TranscriptOct 31 2016, 9:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

bd808 updated the task description. (Show Details)Oct 31 2016, 9:57 PM

let's see if this works

root@tools-bastion-03:~# qconf -de tools-exec-cyberbot.eqiad.wmflabs
root@tools-bastion-03.tools.eqiad.wmflabs removed "tools-exec-cyberbot.eqiad.wmflabs" from execution host list

theory is this was deleted but still returns by /usr/bin/qconf -sel

graphite-labs.wikimedia.org.png (300×600 px, 32 KB)

bd808 mentioned this in T151980: Reduce Precise OGE exec hosts to 10.Dec 1 2016, 9:18 PM

bd808 mentioned this in T154539: Reduce Precise OGE exec hosts to 5.Jan 4 2017, 12:03 AM

	F4682058: graphite-labs.wikimedia.org.png
	Nov 1 2016, 12:00 AM

	F4681676: graphite-labs.wikimedia.org.png
	Oct 31 2016, 9:57 PM

Data collection for tools job_count seems to be brokenClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Data collection for tools job_count seems to be broken
Closed, ResolvedPublic
Actions