So, Josve05a reported on #mediawiki-visualeditor that https://war.wikipedia.org/w/index.php?title=Mansellia_annulata&veaction=edit is one of the failing pages. @Krenair and I confirmed it. I grepped parsoid logs and as you can see, the serialization requests never even got to Parsoid. So, the 400 is being returned further up in the request pipeline. This doesn't seem Parsoid-related.

subbu@earth:~$ for wtp in `ssh ssastry@bast1001.wikimedia.org cat /etc/dsh/group/parsoid` ; do echo $wtp ; ssh ssastry@$wtp grep warwiki /var/log/parsoid/parsoid.log | grep serializ ; done
wtp1001.eqiad.wmnet
{"name":"parsoid","hostname":"wtp1001","pid":9437,"level":30,"logType":"info","wiki":"warwiki","title":"Whitfieldia_elongata","oldId":null,"reqId":"9d16f697-58ad-11e5-ab88-e2dd9c7b3958","msg":"started serializing","longMsg":"started serializing","time":"2015-09-11T17:50:39.426Z","v":0}
{"name":"parsoid","hostname":"wtp1001","pid":9437,"level":30,"logType":"info","wiki":"warwiki","title":"Whitfieldia_elongata","oldId":6198805,"reqId":"9d16f697-58ad-11e5-ab88-e2dd9c7b3958","msg":"completed serializing in 219 ms","longMsg":"completed serializing in 219 ms","time":"2015-09-11T17:50:39.525Z","v":0}
wtp1002.eqiad.wmnet
wtp1003.eqiad.wmnet
wtp1004.eqiad.wmnet
wtp1005.eqiad.wmnet
wtp1006.eqiad.wmnet
wtp1007.eqiad.wmnet
wtp1008.eqiad.wmnet
wtp1009.eqiad.wmnet
{"name":"parsoid","hostname":"wtp1009","pid":22819,"level":30,"logType":"info","wiki":"warwiki","title":"Gumaramit:Krenair_(WMF)/Sandbox","oldId":null,"reqId":"6b2baded-58ae-11e5-9f3c-a1707e2935e0","msg":"started serializing","longMsg":"started serializing","time":"2015-09-11T17:56:25.039Z","v":0}
{"name":"parsoid","hostname":"wtp1009","pid":22819,"level":30,"logType":"info","wiki":"warwiki","title":"Gumaramit:Krenair_(WMF)/Sandbox","oldId":6198807,"reqId":"6b2baded-58ae-11e5-9f3c-a1707e2935e0","msg":"completed serializing in 52 ms","longMsg":"completed serializing in 52 ms","time":"2015-09-11T17:56:25.087Z","v":0}
wtp1010.eqiad.wmnet
wtp1011.eqiad.wmnet
wtp1012.eqiad.wmnet
wtp1013.eqiad.wmnet
wtp1014.eqiad.wmnet
wtp1015.eqiad.wmnet
{"name":"parsoid","hostname":"wtp1015","pid":6321,"level":30,"logType":"info","wiki":"warwiki","title":"Gumaramit:Krenair_(WMF)/Sandbox","oldId":null,"reqId":"62fd1b0c-58ae-11e5-89e6-81926e3d9e44","msg":"started serializing","longMsg":"started serializing","time":"2015-09-11T17:56:11.470Z","v":0}
{"name":"parsoid","hostname":"wtp1015","pid":6321,"level":30,"logType":"info","wiki":"warwiki","title":"Gumaramit:Krenair_(WMF)/Sandbox","oldId":null,"reqId":"62fd1b0c-58ae-11e5-89e6-81926e3d9e44","msg":"completed serializing in 168 ms","longMsg":"completed serializing in 168 ms","time":"2015-09-11T17:56:11.474Z","v":0}
wtp1016.eqiad.wmnet
wtp1017.eqiad.wmnet
wtp1018.eqiad.wmnet
wtp1019.eqiad.wmnet
wtp1020.eqiad.wmnet
wtp1021.eqiad.wmnet
wtp1022.eqiad.wmnet
wtp1023.eqiad.wmnet
{"name":"parsoid","hostname":"wtp1023","pid":31228,"level":30,"logType":"info","wiki":"warwiki","title":"Syahan nga Pakli","oldId":null,"reqId":null,"msg":"started serializing","longMsg":"started serializing","time":"2015-09-11T18:24:21.809Z","v":0}
{"name":"parsoid","hostname":"wtp1023","pid":31228,"level":30,"logType":"info","wiki":"warwiki","title":"Syahan nga Pakli","oldId":null,"reqId":null,"msg":"completed serializing in 90 ms","longMsg":"completed serializing in 90 ms","time":"2015-09-11T18:24:21.811Z","v":0}
{"name":"parsoid","hostname":"wtp1023","pid":31228,"level":30,"logType":"info","wiki":"warwiki","title":"Syahan nga Pakli","oldId":null,"reqId":null,"msg":"started serializing","longMsg":"started serializing","time":"2015-09-11T18:26:01.007Z","v":0}
{"name":"parsoid","hostname":"wtp1023","pid":31228,"level":30,"logType":"info","wiki":"warwiki","title":"Syahan nga Pakli","oldId":null,"reqId":null,"msg":"completed serializing in 8 ms","longMsg":"completed serializing in 8 ms","time":"2015-09-11T18:26:01.008Z","v":0}
wtp1024.eqiad.wmnet

Issue in RESTBase then?

It turns out that this is triggered by really old HTML revisions (rendered and stored in early March) not having a mw:TimeUuid meta tag in the head. A newly added sanity check now detects this issue, and refuses to guess in order to avoid corruptions. It only affects stored renders that haven't been touched since, or where re-renders have not changed the content at all.

A temporary work-around is to edit the page once using the wikitext editor.

In the meantime, we will make sure to re-render all HTML revisions stored before early March.

Thanks!

I have started a script to replace all renders from before April 1st, which is when the change to insert the mw:TimeUuid meta tag was deployed.

To speed things up a little bit, I have now increased the parallelism of re-renders by performing them asynchronously to old revision deletions:

This is running in a screen on restbase1002. Current token range being worked on is -8454960161286145256 out of a total of about -9254960161286145256 to 9254960161286145256, meaning that even at this pace it'll take a while to complete.

Progress update: The script is at -3951775668321071380, so a few more days to go.

Thanks a lot for your work and for the update! It's much appreciated when we can give an ETA :)

We are now getting close to 60%. Based on progress so far, the refresh of all old renders should finish less than 48 hours from now.

From the en.wiki help desk: https://en.wikipedia.org/w/index.php?title=Wikipedia:Help_desk&diff=682483785&oldid=682482046

For the past several days, whenever I have tried to make anything more than a simple change using the "Edit" tool (rather than "Edit Source"), I have been unable to save my work due to a "parsoidserver-http: HTTP 400" error.

In T112286#1654479, @GWicke wrote:

The refresh script finished on Wednesday, so this issue should now be resolved. Thanks for your patience!

In T112286#1669264, @NQ wrote:

From the en.wiki help desk: https://en.wikipedia.org/w/index.php?title=Wikipedia:Help_desk&diff=682483785&oldid=682482046

For the past several days, whenever I have tried to make anything more than a simple change using the "Edit" tool (rather than "Edit Source"), I have been unable to save my work due to a "parsoidserver-http: HTTP 400" error.

Could we get some specific information? For which page that occurred? Which revision? And, if possible, provide the time you attempted the edit.

Looking at the list of thrown 4xx errors for transforms I'm thinking this is partially caused by T113117: RESTBase wrongly refusing to serve revision text if username is hidden, so will put it as a blocker.

Another interesting and possibly-related task is T113163: Cannot save existing pages using IE11, but it claims that error can be witnessed only in IE11, but not in other browsers.

No new reports for 20 days now. Are we out of the woods for this one? If so, I'd like to remove RB's double logging of 4xx Parsoid errors.

I've recently seen reports of failures without a code, and reports of 500 errors from non-WMF wikis, but I think those are not relevant here.

1. Go to https://en.wikipedia.org/wiki/Jonathan_Mendelsohn.
2. Click on the redlink labeled 'Nicola' in the final paragraph.
3. Click on 'Create'.

I reliably get an HTTP 400 error this way.

The request is an HTTP POST to https://en.wikipedia.org/api/rest_v1/transform/wikitext/to/html/Nicola_Mendelsohn/0 with form data:

The response is:

{"type":"https://restbase.org/errors/invalid_request","method":"post","detail":"Missing request parameter: wikitext","uri":"/en.wikipedia.org/v1/transform/wikitext/to/html/Nicola_Mendelsohn/0"}

@ori, it looks like VE has started to use the new stash API when switching from wikitext to visual editing. That API mode did not exist at the time of the original report, so this is definitely unrelated.

I think it would be nicer to relax our input validation to accept the empty string as an input there, as it would save clients from implementing a special case for blank pages. I have created T118070 to track this.