Monitor bigbrother
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	scfc
	Feb 26 2015, 8:17 AM

Description

I only noticed the syntax error in bigbrother when I logged into tools-submit and looked at /var/log/syslog:

Feb 26 06:17:57 tools-submit kernel: [128497.321239] init: bigbrother main process (29839) terminated with status 255
Feb 26 06:17:57 tools-submit kernel: [128497.321265] init: bigbrother respawning too fast, stopped

So it would be nice if it is monitored that the service is actually up and running.

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T90534 Make toolforge reliable enough (tracking)
Invalid	None	T90845 Set up sufficient monitoring for toollabs
Declined	None	T91414 make bigbrother or its replacement reliable
Declined	None	T90850 Monitor bigbrother

Event Timeline

scfc created this task.Feb 26 2015, 8:17 AM

scfc raised the priority of this task from to Needs Triage.

scfc updated the task description. (Show Details)

scfc added projects: Toolforge, Cloud-Services.

scfc added subscribers: yuvipanda, Aklapper, Pietrodn and 9 others.

Steinsplitter unsubscribed.Feb 26 2015, 9:04 AM

Steinsplitter subscribed.

Hmm, I wonder how exactly we would do this, since we don't really have active checks atm.

JanZerebecki added a parent task: T91414: make bigbrother or its replacement reliable.Mar 3 2015, 5:17 PM

yuvipanda added a project: ToolLabs-Goals-Q4.Mar 25 2015, 9:20 PM

yuvipanda moved this task from Backlog to Measurement / Monitoring on the ToolLabs-Goals-Q4 board.Mar 25 2015, 9:43 PM

scfc triaged this task as Lowest priority.Apr 6 2015, 11:13 AM

scfc moved this task from Backlog to Ready to be worked on on the Toolforge board.

yuvipanda removed a project: ToolLabs-Goals-Q4.Jun 24 2015, 4:51 PM

yuvipanda set Security to None.

Luke081515 moved this task from Triage to Backlog on the Cloud-Services board.Mar 25 2016, 4:11 PM

The timestamp on the scoreboard file should change on each pass through bigbrother.py's run loop. Can we setup an icinga alert if that timestamp is more than N minutes old?

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:54 PM

https://en.wikipedia.org/wiki/Quis_custodiet_ipsos_custodes%3F

bd808 mentioned this in T208357: toolforge - Deprecate BigBrother in Grid Engine.Oct 31 2018, 7:50 PM

BigBrother has been disabled in Toolforge.

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Bigbrother_(Deprecated)

• GTirloni closed this task as Declined.Dec 12 2018, 8:55 AM

• GTirloni edited projects, added cloud-services-team (Kanban); removed Cloud-Services.