Maniphest T92614

Network port saturated for labstore1001
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	yuvipanda
	Mar 13 2015, 11:28 AM

Tags

Referenced Files

None

Subscribers

Description

https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1426245937.7&target=servers.labstore1001.network.bond0.rx_byte.value&target=servers.labstore1001.network.bond0.tx_byte.value has data (ganglia doesn't again). My bet is on some tool maxing out NFS bandwidth.

Related Objects

Mentioned In: T122638: GridEngine down due to bdb issues
T92618: Ganglia broken for labstore1001 (again)

Event Timeline

yuvipanda created this task.Mar 13 2015, 11:28 AM

yuvipanda raised the priority of this task from to Unbreak Now!.

yuvipanda updated the task description. (Show Details)

yuvipanda added projects: Cloud-Services, acl*sre-team.

yuvipanda added subscribers: yuvipanda, coren, faidon, mark.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2015, 11:28 AM

faidon mentioned this in T92618: Ganglia broken for labstore1001 (again).Mar 13 2015, 12:10 PM

Turns out it was a tool that had started up a thousand or so jobs that all hit NFS, saturating everything. I've killed all the jobs, and will notify the bot authors soon.

Also, we need to have an alert for network utilization on labstores.

valhallasw mentioned this in T122638: GridEngine down due to bdb issues.Dec 30 2015, 11:19 AM

• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:53 PM

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptJun 7 2017, 6:53 PM