Across all containers on eqiad for example:
27513 thumbor-ip-10.64.32.134 27523 thumbor-ip-10.64.48.229 27905 thumbor-ip-10.64.16.189 28460 thumbor-ip-10.64.32.135
While mediawiki uses poolcounter for rate limiting internal IPs, Thumbor is in theory supposed to use it only for external IPs. It's fairly clear that issues like T338765 are being caused by this as when a request for a private wiki thumbnail is rejected, the error message includes a Kubernetes worker IP address key used to check poolcounter.
I would propose that we add an mechanism for excluding rate limiting internal IP addresses unless we want to keep this behaviour for internal IPs. Should we be using x-client-ip instead of x-forwarded-for?
It's worth noting that Thumbor explicitly uses X-Forwarded-For for this purpose (splitting it on commas and selecting the first element), and so something very odd is happening with these requests given that all 4 IPs are Kubernetes hosts.
All of the Kubernetes hosts above are running changeprop, but that's not a guaranteed link as most kubernetes workers are. However, lots of requests from changeprop seem likely in light of T337649.