eqiad squid performances issue
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ayounsi
	Mar 16 2020, 2:49 PM

Description

Originally guessed in T245121, more visibility has been added with T245176.

I created a quick dashboard: https://grafana.wikimedia.org/d/i5YA-BXWz/squid?orgId=1
But it looks like the Prometheus exporter in eqiad is often taking a lot of time to reply (eg 90s).

ayounsi@prometheus1003:~$ time curl install1003.wikimedia.org:9301/metrics -s | grep _up
# HELP squid_up Was the last query of squid successful?
# TYPE squid_up gauge
squid_up{host="localhost"} 1

real	1m34.694s
user	0m0.012s
sys	0m0.008s

The same issue doesn't happen in codfw.

The amount of requests is quite small ~8rps. So I'd think there is a miss-configuration somewhere?

Details

	Project	Branch	Lines +/-	Subject
	operations/puppet	production	+39 -30	squid3: bump max open file descriptors

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		fgiunchedi	T247759 eqiad squid performances issue
		Resolved		Peter	T247586 WebPageTest and WebPageReplay data unreachable in Grafana

Event Timeline

ayounsi triaged this task as High priority.Mar 16 2020, 2:49 PM

ayounsi created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 16 2020, 2:49 PM

ayounsi mentioned this in T245176: Add Prometheus Squid exporter.Mar 16 2020, 2:50 PM

ayounsi mentioned this in T224576: Upgrade install servers to Buster.Mar 16 2020, 3:05 PM

jbond added a subscriber: jbond.Mar 16 2020, 4:44 PM

hashar added a subscriber: hashar.Mar 16 2020, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2020-03-17T10:20:01Z] <godog> bounce squid on install1003 T247759

Change 580296 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] squid3: bump max open file descriptors

https://gerrit.wikimedia.org/r/580296

gerritbot added a project: Patch-For-Review.Mar 17 2020, 10:30 AM

CDanis added a subtask: T247586: WebPageTest and WebPageReplay data unreachable in Grafana.Mar 17 2020, 10:51 AM

I've bumped the limits for squid on install1003 and things look good now, the permanent fix is in https://gerrit.wikimedia.org/r/580296

In T224576#5972281, @hashar wrote:

When building a docker container on contint1001.wikimedia.org with docker-pkg, pip gets proxy timeout error when using http://webproxy.eqiad.wmnet:8080.

I have manually switched to the codfw one (http://webproxy.codfw.wmnet:8080) and it worked fine.

So I guess install1003.wikimedia.org has an issue of some sort?

I have triggered a build for that container and this time it worked all fine. So it seems install1003 Squid now behave properly :) Thank you!

Change 580296 merged by Filippo Giunchedi:
[operations/puppet@production] squid3: bump max open file descriptors

https://gerrit.wikimedia.org/r/580296