Maniphest T192713

labstore1003 load spikes
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• chasemp
	Apr 21 2018, 12:48 PM

Description

We have been paged 3 times for this and it resolves before I can see what's up.

At the moment I see the following serious users:

root@tools-bastion-03:~# host 10.68.20.221
221.20.68.10.in-addr.arpa domain name pointer mwoffliner4.mwoffliner.eqiad.wmflabs.
root@tools-bastion-03:~# host 10.68.16.103
103.16.68.10.in-addr.arpa domain name pointer maps-tiles3.maps.eqiad.wmflabs.
root@tools-bastion-03:~# host 10.68.16.70
70.16.68.10.in-addr.arpa domain name pointer maps-wma1.maps.eqiad.wmflabs.

But none of them are doing anything abusive afaict.

Related Objects

Mentioned Here: T187456: Decommission labstore100[123] and their disk shelves

Event Timeline

• chasemp created this task.Apr 21 2018, 12:48 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 21 2018, 12:48 PM

• chasemp triaged this task as Medium priority.Apr 21 2018, 12:48 PM

I jumped in on mwoffliner4.mwoffliner.eqiad.wmflabs and hot patched it to:

modules='act_mirr ifb'
nfs_write='6000kbps'
nfs_read='700kbps'
nfs_dumps_read='800kbps'
egress='30000kbps'
iface='eth0'

Then ran (idempotent) /usr/local/sbin/tc-setup as that was the only instance I could see at the time thrashing IO. We didn't get paged again after this and the 3.

For the record, this paged again today 2018-05-04 (flapping)

Some graph data.

eth0 RX/TX bytes:

https://graphite.wikimedia.org/S/B

load avg:

https://graphite.wikimedia.org/S/C

I get an error trying to run nethogs:

# nethogs eth0
creating socket failed while establishing local IP - are you root?

Bstorm added a subscriber: Bstorm.May 4 2018, 3:20 PM

Bstorm added a project: cloud-services-team (Kanban).Oct 18 2018, 4:49 PM

It is now replaced by cloudstore1008/9 T187456

labstore1003 load spikesClosed, DeclinedPublicActions

Description

Related Objects

Event Timeline

labstore1003 load spikes
Closed, DeclinedPublic
Actions