Page MenuHomePhabricator

Why do we have 2 sets of squid proxies?
Closed, ResolvedPublic

Description

As long as I can remember we have had 2 separate sets of squid proxies.

One is url-downloader on a dedicated pair of Ganeti VMs, one in eqiad, one in codfw.

The others are webproxy on installservers, also Ganeti VMs, also one in eqiad and one in codfw.

Does anyone know why we have them both and don't just use a single one?

Do we really need them both?

Event Timeline

Dzahn created this task.May 29 2020, 1:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 29 2020, 1:54 PM

I recall seeing that url-downloader has some download limits, and some server-side upload requests didn't come through for that reason. Not sure if there is a usecase for limiting that through.

I don't have an "official" answer, but I always treated them as follows:

  • url-downloader => the proxy for applications/services to reach resources on the public internet
  • webproxy => the proxies the machines themselves use to reach out to specific public resources like debian security packages, npm/pypi/gems repos/packages (e.g. for CI), openstreetmap things.

Historically speaking, and in my understanding as well as https://wikitech.wikimedia.org/wiki/Url-downloader, the idea of url-downloader was to be used by mediawiki, it's grown to serve other services as well (e.g. citoid, cxserver).

A differentiating fact in their configuration is also the fact that url-downloader does not cache anything, whereas webproxy does (as it makes sense to not download 1500 times the exact same deb package from the internet).

I am not aware of any documentation of the above however and others experiences might be different. I am also pretty sure we have mistakes and grey areas.

Dzahn triaged this task as Medium priority.Jun 4 2020, 9:19 AM

@Dzahn good to close after Alex's answer?

Dzahn closed this task as Resolved.Oct 20 2020, 9:23 PM
Dzahn claimed this task.

Thanks for the detailed answer @akosiaris

Up to the "different config" part I would have thought about unifying them. But if there are differences in use cases as you explained then let's not worry about it, don't currently see a big advantage in changing it and it would mean having to check quite a few things to make sure nothing breaks.

@Marostegui Yea, thanks for the ping. After consideration, yes, closing it. The question was answered :)