So, if we can track down which MediaWiki commit started to introduce "set-cookie" headers (instead of "Set-Cookie"), we could hopefully submit a patch to capitalize it.

The root of the problem lies in the fact that Wikidata-Toolkit uses a very low-level Java API to make HTTP requests, which is therefore quite brittle.

This seems related to T249526.

There was no change to MediaWiki with respect to output of Set-Cookie headers. For that matter, MediaWiki does not directly output the Set-Cookie headers at all, it simply calls PHP's setcookie function and PHP handles the output.

Further, the headers output from PHP itself are in fact using the "Set-Cookie" casing:

anomie@mwdebug1002:~$ curl -v -H 'Host: it.wikipedia.org' "http://$(hostname -i)/w/api.php?action=query&meta=tokens&type=login&format=json"; echo
*   Trying 10.64.0.46...
* TCP_NODELAY set
* Connected to 10.64.0.46 (10.64.0.46) port 80 (#0)
> GET /w/api.php?action=query&meta=tokens&type=login&format=json HTTP/1.1
> Host: it.wikipedia.org
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 08 Apr 2020 13:03:42 GMT
< Server: mwdebug1002.eqiad.wmnet
< X-Powered-By: PHP/7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1
< X-Content-Type-Options: nosniff
< P3P: CP="See https://it.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
< X-Analytics: ns=-1;special=Badtitle
< X-Frame-Options: DENY
< Content-Disposition: inline; filename=api-result.json
< Cache-Control: private, must-revalidate, max-age=0
< Vary: Accept-Encoding
< Set-Cookie: itwikiSession=REDACTED; path=/; HttpOnly
< Set-Cookie: forceHTTPS=true; path=/; HttpOnly
< Backend-Timing: D=122696 t=1586351022770756
< Transfer-Encoding: chunked
< Content-Type: application/json; charset=utf-8

But as noted, the request hitting the public IP rather than the internal appserver do return the headers in lowercase

anomie@mwdebug1002:~$ curl --http1.1 -v "https://it.wikipedia.org/w/api.php?action=query&meta=tokens&type=login&format=json"; echo
*   Trying 2620:0:861:ed1a::1...
* TCP_NODELAY set
* Connected to it.wikipedia.org (2620:0:861:ed1a::1) port 443 (#0)
[... bunch of TLS debugging info ...]
*  SSL certificate verify ok.
> GET /w/api.php?action=query&meta=tokens&type=login&format=json HTTP/1.1
> Host: it.wikipedia.org
> User-Agent: curl/7.52.1
> Accept: */*
> X-Wikimedia-Debug: backend=mwdebug1002.eqiad.wmnet
> 
< HTTP/1.1 200 OK
< date: Wed, 08 Apr 2020 13:13:43 GMT
< server: mwdebug1002.eqiad.wmnet
< x-powered-by: PHP/7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1
< x-content-type-options: nosniff
< p3p: CP="See https://it.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
< x-frame-options: DENY
< content-disposition: inline; filename=api-result.json
< cache-control: private, must-revalidate, max-age=0
< vary: Accept-Encoding
< set-cookie: itwikiSession=REDACTED; path=/; secure; HttpOnly
< set-cookie: forceHTTPS=true; path=/; HttpOnly
< backend-timing: D=74049 t=1586351623859249
< content-type: application/json; charset=utf-8
< x-envoy-upstream-service-time: 74
< X-ATS-Timestamp: 1586351623
< X-Varnish: 497730494
< Age: 0
< X-Cache: cp1081 pass, cp1079 pass
< X-Cache-Status: pass
< Server-Timing: cache;desc="pass"
< Strict-Transport-Security: max-age=106384710; includeSubDomains; preload
< Set-Cookie: WMF-Last-Access=08-Apr-2020;Path=/;HttpOnly;secure;Expires=Sun, 10 May 2020 12:00:00 GMT
< Set-Cookie: WMF-Last-Access-Global=08-Apr-2020;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Sun, 10 May 2020 12:00:00 GMT
< X-Client-IP: 2620:0:861:101:10:64:0:46
< Set-Cookie: GeoIP=US:::37.75:-97.82:v4; Path=/; secure; Domain=.wikipedia.org
< Accept-Ranges: bytes
< Content-Length: 100
< Connection: keep-alive

On that basis, I'm going to poke this to Traffic. My vaguely-informed guess would be that something has recently started to use HTTP/2 in communication between the front and back ends, and HTTP/2 of course requires header field names be transmitted in lowercase (RFC 7540 § 8.1.2).

But as you noted, there's nothing actually wrong with the response. According to both RFC 2616 (HTTP/1.1, June 1999) and RFC 1945 (HTTP/1.0, May 1996), header field names are case insensitive. I'll leave it to them to decide whether to close this as Invalid on that basis, or to try to munge things to output "Set-Cookie" instead of "set-cookie" for the benefit of non-RFC-compliant clients.

Thank you very much for the investigation.

We have patched Wikidata-Toolkit, released a new version, and we will push the fix downstream. We will also be working on using a proper HTTP library which handles these sort of details for us.

I would still be extremely grateful if the header could be reverted to its original capitalization (since OpenRefine is a tool that our users install locally, not a hosted service, rolling out fixes like this has a high cost), but I totally understand if that is not deemed worth doing on WMF's side.

This seems likely related to us switching from using nginx as our internal TLS terminator for the API servers, to using Envoy for that purpose (switchover performed April 6th). As @Anomie guessed, Envoy uses HTTP/2 semantics internally, which is likely the cause of the header being squished to lowercase.

It seems possible that we could add a workaround on our edge CDN to re-translate the header to the expected capitalization, but it isn't something we'd like to maintain forever.

Sure, it's your call. I am sure you have more important things to do than this sort of hack.

As far as I can see, apache (which sits beyond envoy) emits all cookies capitalized. So I am inclined to "fix" this by switching on headers capitalization for http 1.1 in envoy.

@Pintoch I think though we should set a deprecation for the current version of OpenRefine, does that make sense to you?

Switching on headers capitalization would be absolutely fantastic.

We have already published a snapshot of OpenRefine with a hotfix for this issue. We hope that users who are affected by the issue will have the idea of going to the user mailing list or Wikidata, where they will be able to discover this fixed snapshot. But realistically, most people will not upgrade soon - many people are still using obsolete versions released years ago, even if the start page of the tool automatically checks for new stable versions and proposes to upgrade.

So yes, if it is not too hard for you to turn on capitalization on your side, it would be massively useful for us. And for a lot of other projects too, I suspect (anyone using Wikidata-Toolkit, other tools such as VicinoUploader which is affected too).

Change 587697 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] tlsproxy: allow capitalizing headers when connections are http/1.1

https://gerrit.wikimedia.org/r/587697

Change 587697 merged by Giuseppe Lavagetto:
[operations/puppet@production] tlsproxy: allow capitalizing headers when connections are http/1.1

https://gerrit.wikimedia.org/r/587697

Change 587773 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] envoyproxy::tls_terminator: move option for non-SNI to the correct position

https://gerrit.wikimedia.org/r/587773

Change 587773 merged by Giuseppe Lavagetto:
[operations/puppet@production] envoyproxy::tls_terminator: move option for non-SNI to the correct position

https://gerrit.wikimedia.org/r/587773

@Pintoch the behaviour should be restored now. Can you confirm old versions of OpenRefine work correctly now?

Thanks a million, this is very kind of you!

I can confirm this works, edits are coming through again from the latest stable version (unpatched):
https://www.wikidata.org/w/index.php?hidebots=1&hidecategorization=1&tagfilter=openrefine-3.3&limit=50&days=7&title=Special:RecentChanges&urlversion=2

Just to mention that apache httpd does a camel casing by default when proxying back from an http2-talking backend to an http1.x-talking frontend:

https://github.com/icing/mod_h2/blob/master/mod_http2/h2_proxy_session.c#L424

So doing such a transformation, though unnecessary as the RFC says, is definitely not something unheard of.