Page MenuHomePhabricator

Clients failing API login due to dependence on "Set-Cookie" header name casing
Closed, ResolvedPublic

Description

Over the past 48 hours, there seems to have been a change in the behaviour of MediaWiki login API for Wikidata. This causes all logins from Wikidata-Toolkit (Java library) to fail. Among others, this affects OpenRefine users.

The bug comes from the fact that Wikidata started returning cookies using a set-cookie header, whereas Wikidata-Toolkit expects Set-Cookie. HTTP header keys are case insensitive so Wikidata-Toolkit should recognize set-cookie as well, but as things stand this breaks login in all clients relying on this library. See this pull request on Wikidata-Toolkit for details.

While we can of course fix things on Wikidata-Toolkit's side, it would be great if the change on Wikidata's side could be reverted at least temporarily, since fixing this on our side requires making new releases of Wikidata-Toolkit and OpenRefine.

OR issue: https://github.com/OpenRefine/OpenRefine/issues/2526
WDTK issue: https://github.com/Wikidata/Wikidata-Toolkit/issues/491

Event Timeline

Pintoch created this task.Apr 8 2020, 5:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2020, 5:50 AM
Afkbrb awarded a token.Apr 8 2020, 6:05 AM
Afkbrb added a subscriber: Afkbrb.
Pintoch updated the task description. (Show Details)Apr 8 2020, 6:48 AM

So, if we can track down which MediaWiki commit started to introduce "set-cookie" headers (instead of "Set-Cookie"), we could hopefully submit a patch to capitalize it.

The root of the problem lies in the fact that Wikidata-Toolkit uses a very low-level Java API to make HTTP requests, which is therefore quite brittle.

Restricted Application added a project: Platform Engineering. · View Herald TranscriptApr 8 2020, 7:35 AM
Pintoch updated the task description. (Show Details)Apr 8 2020, 7:43 AM
Pintoch added a comment.EditedApr 8 2020, 8:41 AM

This seems related to T249526.

Anomie renamed this task from Wikidata password login via MediaWiki API fails to Clients failing API login due to dependence on "Set-Cookie" header name casing.EditedApr 8 2020, 1:30 PM
Anomie added a subscriber: Anomie.

There was no change to MediaWiki with respect to output of Set-Cookie headers. For that matter, MediaWiki does not directly output the Set-Cookie headers at all, it simply calls PHP's setcookie function and PHP handles the output.

Further, the headers output from PHP itself are in fact using the "Set-Cookie" casing:

anomie@mwdebug1002:~$ curl -v -H 'Host: it.wikipedia.org' "http://$(hostname -i)/w/api.php?action=query&meta=tokens&type=login&format=json"; echo
*   Trying 10.64.0.46...
* TCP_NODELAY set
* Connected to 10.64.0.46 (10.64.0.46) port 80 (#0)
> GET /w/api.php?action=query&meta=tokens&type=login&format=json HTTP/1.1
> Host: it.wikipedia.org
> User-Agent: curl/7.52.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 08 Apr 2020 13:03:42 GMT
< Server: mwdebug1002.eqiad.wmnet
< X-Powered-By: PHP/7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1
< X-Content-Type-Options: nosniff
< P3P: CP="See https://it.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
< X-Analytics: ns=-1;special=Badtitle
< X-Frame-Options: DENY
< Content-Disposition: inline; filename=api-result.json
< Cache-Control: private, must-revalidate, max-age=0
< Vary: Accept-Encoding
< Set-Cookie: itwikiSession=REDACTED; path=/; HttpOnly
< Set-Cookie: forceHTTPS=true; path=/; HttpOnly
< Backend-Timing: D=122696 t=1586351022770756
< Transfer-Encoding: chunked
< Content-Type: application/json; charset=utf-8

But as noted, the request hitting the public IP rather than the internal appserver do return the headers in lowercase

anomie@mwdebug1002:~$ curl --http1.1 -v "https://it.wikipedia.org/w/api.php?action=query&meta=tokens&type=login&format=json"; echo
*   Trying 2620:0:861:ed1a::1...
* TCP_NODELAY set
* Connected to it.wikipedia.org (2620:0:861:ed1a::1) port 443 (#0)
[... bunch of TLS debugging info ...]
*  SSL certificate verify ok.
> GET /w/api.php?action=query&meta=tokens&type=login&format=json HTTP/1.1
> Host: it.wikipedia.org
> User-Agent: curl/7.52.1
> Accept: */*
> X-Wikimedia-Debug: backend=mwdebug1002.eqiad.wmnet
> 
< HTTP/1.1 200 OK
< date: Wed, 08 Apr 2020 13:13:43 GMT
< server: mwdebug1002.eqiad.wmnet
< x-powered-by: PHP/7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1
< x-content-type-options: nosniff
< p3p: CP="See https://it.wikipedia.org/wiki/Special:CentralAutoLogin/P3P for more info."
< x-frame-options: DENY
< content-disposition: inline; filename=api-result.json
< cache-control: private, must-revalidate, max-age=0
< vary: Accept-Encoding
< set-cookie: itwikiSession=REDACTED; path=/; secure; HttpOnly
< set-cookie: forceHTTPS=true; path=/; HttpOnly
< backend-timing: D=74049 t=1586351623859249
< content-type: application/json; charset=utf-8
< x-envoy-upstream-service-time: 74
< X-ATS-Timestamp: 1586351623
< X-Varnish: 497730494
< Age: 0
< X-Cache: cp1081 pass, cp1079 pass
< X-Cache-Status: pass
< Server-Timing: cache;desc="pass"
< Strict-Transport-Security: max-age=106384710; includeSubDomains; preload
< Set-Cookie: WMF-Last-Access=08-Apr-2020;Path=/;HttpOnly;secure;Expires=Sun, 10 May 2020 12:00:00 GMT
< Set-Cookie: WMF-Last-Access-Global=08-Apr-2020;Path=/;Domain=.wikipedia.org;HttpOnly;secure;Expires=Sun, 10 May 2020 12:00:00 GMT
< X-Client-IP: 2620:0:861:101:10:64:0:46
< Set-Cookie: GeoIP=US:::37.75:-97.82:v4; Path=/; secure; Domain=.wikipedia.org
< Accept-Ranges: bytes
< Content-Length: 100
< Connection: keep-alive

On that basis, I'm going to poke this to Traffic. My vaguely-informed guess would be that something has recently started to use HTTP/2 in communication between the front and back ends, and HTTP/2 of course requires header field names be transmitted in lowercase (RFC 7540 § 8.1.2).

But as you noted, there's nothing actually wrong with the response. According to both RFC 2616 (HTTP/1.1, June 1999) and RFC 1945 (HTTP/1.0, May 1996), header field names are case insensitive. I'll leave it to them to decide whether to close this as Invalid on that basis, or to try to munge things to output "Set-Cookie" instead of "set-cookie" for the benefit of non-RFC-compliant clients.

Restricted Application added a project: Operations. · View Herald TranscriptApr 8 2020, 1:31 PM

Thank you very much for the investigation.

We have patched Wikidata-Toolkit, released a new version, and we will push the fix downstream. We will also be working on using a proper HTTP library which handles these sort of details for us.

I would still be extremely grateful if the header could be reverted to its original capitalization (since OpenRefine is a tool that our users install locally, not a hosted service, rolling out fixes like this has a high cost), but I totally understand if that is not deemed worth doing on WMF's side.

CDanis added a project: serviceops.EditedApr 8 2020, 1:49 PM
CDanis added subscribers: Joe, CDanis.

This seems likely related to us switching from using nginx as our internal TLS terminator for the API servers, to using Envoy for that purpose (switchover performed April 6th). As @Anomie guessed, Envoy uses HTTP/2 semantics internally, which is likely the cause of the header being squished to lowercase.

It seems possible that we could add a workaround on our edge CDN to re-translate the header to the expected capitalization, but it isn't something we'd like to maintain forever.

Sure, it's your call. I am sure you have more important things to do than this sort of hack.

MoritzMuehlenhoff triaged this task as High priority.Apr 9 2020, 7:40 AM
Joe added a comment.Apr 9 2020, 7:53 AM

As far as I can see, apache (which sits beyond envoy) emits all cookies capitalized. So I am inclined to "fix" this by switching on headers capitalization for http 1.1 in envoy.

@Pintoch I think though we should set a deprecation for the current version of OpenRefine, does that make sense to you?

Pintoch added a comment.EditedApr 9 2020, 8:06 AM

Switching on headers capitalization would be absolutely fantastic.

We have already published a snapshot of OpenRefine with a hotfix for this issue. We hope that users who are affected by the issue will have the idea of going to the user mailing list or Wikidata, where they will be able to discover this fixed snapshot. But realistically, most people will not upgrade soon - many people are still using obsolete versions released years ago, even if the start page of the tool automatically checks for new stable versions and proposes to upgrade.

So yes, if it is not too hard for you to turn on capitalization on your side, it would be massively useful for us. And for a lot of other projects too, I suspect (anyone using Wikidata-Toolkit, other tools such as VicinoUploader which is affected too).

Change 587697 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] tlsproxy: allow capitalizing headers when connections are http/1.1

https://gerrit.wikimedia.org/r/587697

Change 587697 merged by Giuseppe Lavagetto:
[operations/puppet@production] tlsproxy: allow capitalizing headers when connections are http/1.1

https://gerrit.wikimedia.org/r/587697

Change 587773 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] envoyproxy::tls_terminator: move option for non-SNI to the correct position

https://gerrit.wikimedia.org/r/587773

Change 587773 merged by Giuseppe Lavagetto:
[operations/puppet@production] envoyproxy::tls_terminator: move option for non-SNI to the correct position

https://gerrit.wikimedia.org/r/587773

Joe closed this task as Resolved.Apr 9 2020, 4:35 PM
Joe claimed this task.

From my tests, now we get all cookies correctly set:

$ curl --http1.1 -sIL "https://it.wikipedia.org/w/api.php?action=query&meta=tokens&type=login&format=json" | grep -i set-cookie | cut -d: -f1
Set-Cookie
Set-Cookie
Set-Cookie
Set-Cookie
Set-Cookie

# http2
$ curl -sIL "https://it.wikipedia.org/w/api.php?action=query&meta=tokens&type=login&format=json" | grep -i set-cookie | cut -d: -f1
set-cookie
set-cookie
set-cookie
set-cookie
set-cookie
Joe added a comment.Apr 9 2020, 4:36 PM

@Pintoch the behaviour should be restored now. Can you confirm old versions of OpenRefine work correctly now?

Thanks a million, this is very kind of you!

I can confirm this works, edits are coming through again from the latest stable version (unpatched):
https://www.wikidata.org/w/index.php?hidebots=1&hidecategorization=1&tagfilter=openrefine-3.3&limit=50&days=7&title=Special:RecentChanges&urlversion=2

Antigng added a subscriber: Antigng.Jul 8 2020, 3:52 AM

Just to mention that apache httpd does a camel casing by default when proxying back from an http2-talking backend to an http1.x-talking frontend:

https://github.com/icing/mod_h2/blob/master/mod_http2/h2_proxy_session.c#L424

So doing such a transformation, though unnecessary as the RFC says, is definitely not something unheard of.