I thought we do send Retry-After. Can we get a reproducing case? Maybe some proxy drops it? Because I am looking at the throttling code and clearly see producing the Retry-After header.

Yup, that code looks like it is definitely sent.

I guess we need to look at both the nginx proxy and the varnish proxy?
Traffic might know the answer for the varnish layer.

Can we get a reproducing case?

We could try hammering the endpoint?
There is a log line including response headers from the WDQS referenced in T204267#4887839. That is what it sees.

Well, as mentioned in T204267#4865669, we’re currently looking for Retry-After, case-sensitive. And if I remember correctly, nginx usually sends headers in all lowercase. So unless MediaWiki’s MWHttpRequest::getResponseHeaders normalizes retry-after back into Retry-After, we won’t find the header.

Change 485625 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/WikibaseQualityConstraints@master] Look for Retry-After header case-insensitively

https://gerrit.wikimedia.org/r/485625

In T214032#4896183, @Lucas_Werkmeister_WMDE wrote:

Well, as mentioned in T204267#4865669, we’re currently looking for Retry-After, case-sensitive. And if I remember correctly, nginx usually sends headers in all lowercase. So unless MediaWiki’s MWHttpRequest::getResponseHeaders normalizes retry-after back into Retry-After, we won’t find the header.

That isn't the issue here, the client still does not get the header.

This can be seen in the logs on T204267#4887839 which is reported by https://github.com/wikimedia/mediawiki-extensions-WikibaseQualityConstraints/blob/1ef719f5891a616450e4fc21cfd49527a456b7d1/src/ConstraintCheck/Helper/LoggingHelper.php#L298 and dumps all of the headers into the log.

{"responseHeaders":"{\"server\":[\"nginx\\/1.13.6\"],\"date\":[\"Tue, 15 Jan 2019 15:27:10 GMT\"],\"content-type\":[\"application\\/octet-stream\"],\"connection\":[\"close\"],\"cache-control\":[\"no-cache\"]}"

Also getResponseHeaders and getResponseHeader do normalization of the header using the parseHeader method.
So we should actually only ever be looking for lowercase (I'll add this to code review)

Change 485625 merged by jenkins-bot:
[mediawiki/extensions/WikibaseQualityConstraints@master] Fix Retry-After header handling

https://gerrit.wikimedia.org/r/485625

I tried with https://query.wikidata.org/sparql?throttleMe=1&query=SELECT%20*%20WHERE%20%7B%20?x%20?y%20?z%20%7D%20LIMIT%2010 which always produces 429, and indeed I do not see Retry-After. Moreover, I also do not see X-Served-By header too. Looks like something is stripping headers here. Same happens on localhost, so it's probably nginx. I'll dig into nginx configs and try to figure out why.

Change 491870 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/puppet@production] Turn off proxy_intercept_errors for nginx

https://gerrit.wikimedia.org/r/491870

Change 491870 merged by Gehel:
[operations/puppet@production] Turn off proxy_intercept_errors for nginx

https://gerrit.wikimedia.org/r/491870

Patch merged, let me know if the issue isn't fixed.

I don't get any Retry-After related header when I get 429 error in my client. I'm using Python requests library to query Wikidata service.

These are the headers I'm getting: ['Date', 'Server', 'X-Cache', 'X-Cache-Status', 'Server-Timing', 'Strict-Transport-Security', 'Report-To', 'NEL', 'Set-Cookie', 'X-Client-IP', 'Content-Type', 'Content-Length', 'Connection']