Now we have better tooling we can examine this more in depth; NPR requests are actually timing out and responding with 504. This would be slightly weird for a deliberate block as you're more likely to get a 403 and for the response to be relatively fast where indeed these are actually timing out.

https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1M,to:now))&_a=(description:'Dashboard%20for%20citoid%20service%20(service-runner%20service).%20https:%2F%2Fwww.mediawiki.org%2Fwiki%2FCitoid',filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:kubernetes.namespace_name,negate:!f,params:(query:citoid),type:phrase),query:(match_phrase:(kubernetes.namespace_name:citoid))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:outgoingReqResult.hostname.keyword,negate:!f,params:(query:www.npr.org),type:phrase),query:(match_phrase:(outgoingReqResult.hostname.keyword:www.npr.org)))),fullScreenMode:!f,options:(darkTheme:!f,useMargins:!t),query:(language:lucene,query:'*'),timeRestore:!f,title:citoid,viewMode:view)

This is particular bad for our users as there are a lot of requests and they have to wait since it's a timeout rather than a relatively quick 403.

@zoe suggested SSL handshake issue can cause timeouts sometimes?

@akosiaris any ideas whether this might be the case/how to test?

@Mvolz Try this from your test env

curl --http1.1 -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Citoid/WMF (mailto:noc@wikimedia.org)" -vvv https://www.npr.org/2024/03/19/1239528787/female-genital-mutilation-is-illegal-in-the-gambia-but-maybe-not-for-much-longer

While not entirely consistent, the above times out for me after 55 seconds in most cases. Using HTTP2 instead of HTTP/1.1 returns even faster. My guess is that Citoid has been put in some blocklist on their part.