As long as I can remember we have had 2 separate sets of squid proxies.
One is url-downloader on a dedicated pair of Ganeti VMs, one in eqiad, one in codfw.
The others are webproxy on installservers, also Ganeti VMs, also one in eqiad and one in codfw.
Does anyone know why we have them both and don't just use a single one?
Do we really need them both?
I recall seeing that url-downloader has some download limits, and some server-side upload requests didn't come through for that reason. Not sure if there is a usecase for limiting that through.
I don't have an "official" answer, but I always treated them as follows:
Historically speaking, and in my understanding as well as https://wikitech.wikimedia.org/wiki/Url-downloader, the idea of url-downloader was to be used by mediawiki, it's grown to serve other services as well (e.g. citoid, cxserver).
A differentiating fact in their configuration is also the fact that url-downloader does not cache anything, whereas webproxy does (as it makes sense to not download 1500 times the exact same deb package from the internet).
I am not aware of any documentation of the above however and others experiences might be different. I am also pretty sure we have mistakes and grey areas.
Thanks for the detailed answer @akosiaris
Up to the "different config" part I would have thought about unifying them. But if there are differences in use cases as you explained then let's not worry about it, don't currently see a big advantage in changing it and it would mean having to check quite a few things to make sure nothing breaks.
@Marostegui Yea, thanks for the ping. After consideration, yes, closing it. The question was answered :)
Updated docs:
https://wikitech.wikimedia.org/w/index.php?title=Obsolete%3ASquids&type=revision&diff=1885659&oldid=1820051
https://wikitech.wikimedia.org/w/index.php?title=HTTP_proxy&type=revision&diff=1885661&oldid=1859940
https://wikitech.wikimedia.org/w/index.php?title=Url-downloader&action=history