We are seeing some tools having intermittent 104 errors, in durations of several minutes to several wikis at a time.
See children tasks for details.
This is caused by us hitting the hard limit of 500 concurrent connections to the CDN nodes (shared by all the wikis), from a single IP (k8s worker node).
Things to do:
- document that limit somewhere - added a note https://wikitech.wikimedia.org/wiki/Help:Toolforge#Constraints_of_Toolforge
- try to find a way to figure out tools that might be hitting it - will do in another task