Page MenuHomePhabricator

Why do we have 2 sets of squid proxies?
Closed, ResolvedPublic


As long as I can remember we have had 2 separate sets of squid proxies.

One is url-downloader on a dedicated pair of Ganeti VMs, one in eqiad, one in codfw.

The others are webproxy on installservers, also Ganeti VMs, also one in eqiad and one in codfw.

Does anyone know why we have them both and don't just use a single one?

Do we really need them both?

Event Timeline

I recall seeing that url-downloader has some download limits, and some server-side upload requests didn't come through for that reason. Not sure if there is a usecase for limiting that through.

I don't have an "official" answer, but I always treated them as follows:

  • url-downloader => the proxy for applications/services to reach resources on the public internet
  • webproxy => the proxies the machines themselves use to reach out to specific public resources like debian security packages, npm/pypi/gems repos/packages (e.g. for CI), openstreetmap things.

Historically speaking, and in my understanding as well as, the idea of url-downloader was to be used by mediawiki, it's grown to serve other services as well (e.g. citoid, cxserver).

A differentiating fact in their configuration is also the fact that url-downloader does not cache anything, whereas webproxy does (as it makes sense to not download 1500 times the exact same deb package from the internet).

I am not aware of any documentation of the above however and others experiences might be different. I am also pretty sure we have mistakes and grey areas.

Dzahn triaged this task as Medium priority.Jun 4 2020, 9:19 AM
Dzahn claimed this task.

Thanks for the detailed answer @akosiaris

Up to the "different config" part I would have thought about unifying them. But if there are differences in use cases as you explained then let's not worry about it, don't currently see a big advantage in changing it and it would mean having to check quite a few things to make sure nothing breaks.

@Marostegui Yea, thanks for the ping. After consideration, yes, closing it. The question was answered :)