Page MenuHomePhabricator

eqiad squid performances issue
Closed, ResolvedPublic

Description

Originally guessed in T245121, more visibility has been added with T245176.

I created a quick dashboard: https://grafana.wikimedia.org/d/i5YA-BXWz/squid?orgId=1
But it looks like the Prometheus exporter in eqiad is often taking a lot of time to reply (eg 90s).

ayounsi@prometheus1003:~$ time curl install1003.wikimedia.org:9301/metrics -s | grep _up
# HELP squid_up Was the last query of squid successful?
# TYPE squid_up gauge
squid_up{host="localhost"} 1

real	1m34.694s
user	0m0.012s
sys	0m0.008s

The same issue doesn't happen in codfw.

The amount of requests is quite small ~8rps. So I'd think there is a miss-configuration somewhere?

Details

Related Gerrit Patches:
operations/puppet : productionsquid3: bump max open file descriptors

Event Timeline

ayounsi triaged this task as High priority.Mon, Mar 16, 2:49 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Mar 16, 2:49 PM
jbond added a subscriber: jbond.Mon, Mar 16, 4:44 PM
hashar added a subscriber: hashar.Mon, Mar 16, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-17T10:20:01Z] <godog> bounce squid on install1003 T247759

Change 580296 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] squid3: bump max open file descriptors

https://gerrit.wikimedia.org/r/580296

I've bumped the limits for squid on install1003 and things look good now, the permanent fix is in https://gerrit.wikimedia.org/r/580296

When building a docker container on contint1001.wikimedia.org with docker-pkg, pip gets proxy timeout error when using http://webproxy.eqiad.wmnet:8080.
I have manually switched to the codfw one (http://webproxy.codfw.wmnet:8080) and it worked fine.
So I guess install1003.wikimedia.org has an issue of some sort?

I have triggered a build for that container and this time it worked all fine. So it seems install1003 Squid now behave properly :) Thank you!

Change 580296 merged by Filippo Giunchedi:
[operations/puppet@production] squid3: bump max open file descriptors

https://gerrit.wikimedia.org/r/580296

fgiunchedi closed this task as Resolved.Tue, Mar 17, 12:07 PM
fgiunchedi claimed this task.

Fix is deployed, looking good!

Dzahn awarded a token.Tue, Mar 17, 6:59 PM