Just stating for the record that connection refused/reset messages will come from our edge caching layer, specifically from the tcp stack of our servers there, so it wouldn't be related to a migration to kubernetes (which is still only partial, btw).

In T356163#9497689, @Joe wrote:

migration to kubernetes (which is still only partial, btw).

I was talking about the migration of my tool. It's now running on k8s 100%.

@Joe thanks! Yes, the issue is unrelated to the k8s workers, we were just hitting the limit of concurrent connections to the CDN per-ip.

In that sense, @Leloiandudu have you had any issues lately? We found that there was another tool doing many requests in parallel, that might have affected all the tools running on the same worker node (including yours), but the maintainer is working on fixing that, so should be solved soon.

If your tool does requests in parallel, just make sure to do only a few, as we can only have 500 per node, that is shared with many other tools, after that is reached, the CDN will drop all requests for 300s.

Also, reminder in case you were not doing it, to use a proper user-agent (https://meta.wikimedia.org/wiki/User-Agent_policy) when doing requests to the wikis, that will help you avoid getting rate-limited and help the wikis know the requests come from a tool (and which one).

If you are still finding issues, please let me know so I can debug and try to figure out which other tools might be doing many requests in parallel to help them avoid getting everyone blocked 👍

Still getting these from time to time. Last time: 11 Apr 6:15:48 AM (UTC)