Page MenuHomePhabricator

eqiad squid performances issue
Closed, ResolvedPublic


Originally guessed in T245121, more visibility has been added with T245176.

I created a quick dashboard:
But it looks like the Prometheus exporter in eqiad is often taking a lot of time to reply (eg 90s).

ayounsi@prometheus1003:~$ time curl -s | grep _up
# HELP squid_up Was the last query of squid successful?
# TYPE squid_up gauge
squid_up{host="localhost"} 1

real	1m34.694s
user	0m0.012s
sys	0m0.008s

The same issue doesn't happen in codfw.

The amount of requests is quite small ~8rps. So I'd think there is a miss-configuration somewhere?

Event Timeline

ayounsi triaged this task as High priority.Mar 16 2020, 2:49 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 16 2020, 2:49 PM
jbond added a subscriber: jbond.Mar 16 2020, 4:44 PM
hashar added a subscriber: hashar.Mar 16 2020, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-17T10:20:01Z] <godog> bounce squid on install1003 T247759

Change 580296 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] squid3: bump max open file descriptors

I've bumped the limits for squid on install1003 and things look good now, the permanent fix is in

When building a docker container on with docker-pkg, pip gets proxy timeout error when using http://webproxy.eqiad.wmnet:8080.

I have manually switched to the codfw one (http://webproxy.codfw.wmnet:8080) and it worked fine.

So I guess has an issue of some sort?

I have triggered a build for that container and this time it worked all fine. So it seems install1003 Squid now behave properly :) Thank you!

Change 580296 merged by Filippo Giunchedi:
[operations/puppet@production] squid3: bump max open file descriptors

fgiunchedi closed this task as Resolved.Mar 17 2020, 12:07 PM
fgiunchedi claimed this task.

Fix is deployed, looking good!

Dzahn awarded a token.Mar 17 2020, 6:59 PM