Page MenuHomePhabricator

REST API calls suddenly all returning 400
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

Browse to e.g. https://en.wikipedia.org/api/rest_v1/page/html/Veljko_Ra%C5%BEnatovi%C4%87/1212334420?redirect=false

This seems to happen for lots (most? all?) API calls I am doing collecting data for my spamcheck tool.

What happens?:

Instead of the HTML I get the following response:

{"type":"https://mediawiki.org/wiki/HyperSwitch/errors/unknown_error","method":"get","uri":"/en.wikipedia.org/v1/page/html/Veljko_Ra%C5%BEnatovi%C4%87/1212334420"}

What should have happened instead?:
The Parsoid HTML should be returned.

Event Timeline

Change 1009500 had a related patch set uploaded (by Jaime Nuche; author: Daniel Kinzler):

[mediawiki/core@wmf/1.42.0-wmf.21] REST: allow lower-case method names

https://gerrit.wikimedia.org/r/1009500

Change 1009493 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/alerts@master] mw-on-k8s: Add MediaWikiHTTPErrorRatio alert

https://gerrit.wikimedia.org/r/1009493

@Jgiannelos @Jdforrester-WMF I've created a backport patch to unblock the train: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1009500. But it would still need a pair of eyes before I'm comfortable backporting it

Change 1009500 merged by jenkins-bot:

[mediawiki/core@wmf/1.42.0-wmf.21] REST: allow lower-case method names

https://gerrit.wikimedia.org/r/1009500

Change 1009542 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009542

We can put it in mwdebug before merge and check it.

Change 1009544 had a related patch set uploaded (by Jaime Nuche; author: Daniel Kinzler):

[mediawiki/core@wmf/1.42.0-wmf.21] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009544

@daniel thanks, created a version for 1.42.0-wmf.21 too.

We can put it in mwdebug before merge and check it.

@Ladsgroup would you be up to do that? (EDIT: I think all parsoid is in K8s now, so I'm not sure how to go about doing that)

If someone gives me an example of a failure url (the one on the description works now, I guess rollback?) I can try.

If someone gives me an example of a failure url (the one on the description works now, I guess rollback?) I can try.

Note that we can't try this on mwdebug, since the request goes through restbase.

I put the patch in mwdebug2001 and tested it and it doesn't fix it. We need to find a way to test this.

More debugging:

  • I am testing on a local RESTBase instance connected to mediawiki beta (master)
  • Without any changes to the codebase RESTBase will send a request to /w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/pagebundle
  • The response fails with error:
    • 'The GET request method does not accept a request body'
    • It indicates that indeed MW patch makes the method name uppercase
  • If I manually patch the underlying library that makes the request and replace get with GET i don't see any errors

Here are the raw debug requests from my env:

Body is null in both, the only difference I see that the underlying library gzips the lowercase version but doesn't gzip the uppercase version.

I put the patch in mwdebug2001 and tested it and it doesn't fix it. We need to find a way to test this.

restbase isn't going to hit mwdebug2001... would be nice if we had a way to tell restbase to hit a debug server.

Body is null in both, the only difference I see that the underlying library gzips the lowercase version but doesn't gzip the uppercase version.

Perhaps it disables the body for upper-case GET, but since we have been using lower-case get, we have been sending gezipped bodies with get requests?

This workaround looks like it fixes things on my local RESTBase instance:
https://github.com/wikimedia/restbase/pull/1343

Change 1009544 merged by jenkins-bot:

[mediawiki/core@wmf/1.42.0-wmf.21] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009544

Mentioned in SAL (#wikimedia-operations) [2024-03-07T15:48:49Z] <jnuche@deploy2002> Started scap: Backport for [[gerrit:1009544|REST: ignore request body on GET requests (T359509)]]

Mentioned in SAL (#wikimedia-operations) [2024-03-07T15:50:22Z] <jnuche@deploy2002> jnuche: Backport for [[gerrit:1009544|REST: ignore request body on GET requests (T359509)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-03-07T15:59:56Z] <jnuche@deploy2002> Finished scap: Backport for [[gerrit:1009544|REST: ignore request body on GET requests (T359509)]] (duration: 11m 06s)

This looks like its working now if I query restbase:

curl -v  "http://restbase.svc.codfw.wmnet:7233/test.wikipedia.org/v1/page/summary/Dog"

From the parsoid health chart:
40Xs responses are down to the previous rate

image.png (457×1 px, 149 KB)

jnuche lowered the priority of this task from Unbreak Now! to Needs Triage.EditedMar 7 2024, 4:24 PM

The problem seems fixed in prod now. Thank you everyone your your help with this.

Thank you all for investigating! And sorry for causing a mess...

We need a client patch. It's not OK to use lower-case method names.

We need a client patch. It's not OK to use lower-case method names.

Yea, and it's also not OK to send content-length: 0 with a GET request.

But some clients do - in particular, RESTbase does.

Change 1010932 had a related patch set uploaded (by Jgiannelos; author: Daniel Kinzler):

[mediawiki/core@wmf/1.42.0-wmf.22] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1010932

Change 1009542 merged by jenkins-bot:

[mediawiki/core@master] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1009542

Change 1010932 merged by jenkins-bot:

[mediawiki/core@wmf/1.42.0-wmf.22] REST: ignore request body on GET requests

https://gerrit.wikimedia.org/r/1010932

Mentioned in SAL (#wikimedia-operations) [2024-03-14T13:36:30Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Started scap: Backport for [[gerrit:1010932|REST: ignore request body on GET requests (T359509)]]

Mentioned in SAL (#wikimedia-operations) [2024-03-14T13:38:43Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 jgiannelos and lucaswerkmeister-wmde: Backport for [[gerrit:1010932|REST: ignore request body on GET requests (T359509)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-03-14T13:58:37Z] <logmsgbot> lucaswerkmeister-wmde@deploy2002 Finished scap: Backport for [[gerrit:1010932|REST: ignore request body on GET requests (T359509)]] (duration: 22m 06s)

Metrics looks better now after the backport fix:

image.png (1×1 px, 377 KB)

Also the URL from the original report:

jgiannelos@deploy2002:~$ curl -v -o /dev/null "http://restbase.svc.codfw.wmnet:7233/it.wikipedia.org/v1/page/html/Iremel%27"

< HTTP/1.1 200

jgiannelos@deploy2002:~$
Jgiannelos claimed this task.