Page MenuHomePhabricator

404 error when using VisualEditor: apierror-visualeditor-docserver-http
Closed, ResolvedPublic

Description

It was reported on itwiki that VisualEditor errors out when trying to edit a specific page.
Steps to reproduce:

Or alternatively, on the same page

  • Try to switch to source editor
  • You'll get the "apierror-visualeditor-docserver-http" error

The user report they have no VPN or firewall enabled, and this bug only seems to happen for the given page.

Event Timeline

Daimona created this task.Aug 10 2019, 3:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 10 2019, 3:17 PM

Try to made more deleting lines and modify wikilink. The problem cannot reproduce at every single edit, but it appears very frequently.

Izno added a subscriber: Izno.Sep 14 2019, 3:12 PM

Possibly this discussion also demonstrating the issue.

JJMC89 added subscribers: kaldari, Ammarpad.
JJMC89 added a subscriber: JJMC89.
In T232888, @kaldari wrote:

On the English Wikipedia article Hugh C. Robertson, I was trying to save an edit that I made with VisualEditor, but after filling in the edit summary and clicking "Publish changes", I got the following error:


I then clicked "Dismiss", closed the Save dialog, and tried to save the same edit again. Each time, it would give me the same 404 error.
The error came from https://en.wikipedia.org/w/api.php, with the following POST data:

action: visualeditoredit
format: json
formatversion: 2
campaign: 
summary: copyedit
paction: save
errorformat: html
errorlang: en
errorsuselocal: 1
page: Hugh_C._Robertson
oldid: 915473825
basetimestamp: 20190913133954
starttimestamp: 20190913184637
etag: "915473825/f9cb9f80-d62b-11e9-8729-85d04a2e5aa3-df"
html: [...]
token: 68a45603c83eee982d1df6021d85452d5d7be40a \

The exact error data returned was:

{"errors":[{"code":"apierror-visualeditor-docserver-http","html":"HTTP 404","module":"visualeditoredit"}],"docref":"See https://en.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.","servedby":"mw1290"}

After closing the window and starting from scratch, I wasn't able to reproduce the bug again.

JJMC89 renamed this task from VisualEditor cannot save the edit for a specific page: apierror-visualeditor-docserver-http to 404 error when using VisualEditor: apierror-visualeditor-docserver-http.Sep 14 2019, 7:50 PM
JJMC89 removed a subscriber: JJMC89.

Just another instance of the same thing seen by kaldari, above:

action: visualeditoredit
format: json
formatversion: 2
campaign: 
summary: c
watch: 
paction: save
errorformat: html
errorlang: en
errorsuselocal: 1
page: User:RoySmith/sandbox
oldid: 915824451
basetimestamp: 20190915142553
starttimestamp: 20190915142603
etag: "915824451/bb5d8160-d7c4-11e9-87b1-2d30288168bb-df"
html: [...]
token: c8538334105dcb91b1fe954f7a5975cd5d7e49cb \
{"errors":[{"code":"apierror-visualeditor-docserver-http","html":"HTTP 404","module":"visualeditoredit"}],"docref":"See https://en.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.","servedby":"mw1287"}
IndyJr added a comment.EditedSep 16 2019, 9:52 AM

The problem I reported still go on, on random pages, and is not reproducible. The only workaround I found to don't lose job until then, is to copy and paste into a new window that NOT recovers the unsaved work (because otherwise it doesn't save it again with the same error message).

kaldari triaged this task as High priority.Sep 16 2019, 5:25 PM

Boldly marking this High priority since folks are losing edits, and, according to IndyJr, the bug happens "very frequently".

matmarex added a subscriber: matmarex.

Until now we had no idea what was causing those, but in the most recent bug report at T233127 @Waddie96 discovered that this is caused by the "Two column edit conflict" beta feature. Thank you for that! I can confirm that if you have it enabled and have an edit conflict, you get this weird error message instead of the expected one (like below).

I don't know yet why this happens, but now we at least have something we can debug.

Restricted Application added a project: TCB-Team. · View Herald TranscriptSep 17 2019, 11:35 PM

After some more testing, I don't think it's about the two column edit conflict beta feature, but just edit conflicts in general. I also get the issue sometimes when the beta feature is disabled.

The 404 error is generated by RESTBase. When VE requests RESTBase to transform the HTML to wikitext, we need to pass along the page title, revision ID, and an "etag" (an unique ID for the parsed document). We receive that etag from RESTBase itself when requesting the page HTML for editing. When trying to transform new HTML to wikitext later, RESTBase sometimes returns the 404 error, as if the document it gave us earlier suddenly disappeared.

I don't think it's edit conflict related. I've seen this several times when there was no edit conflict. For example, the one I cited above that happened on User:RoySmith/sandbox. For sure, there was no edit conflict on my sandbox.

At one point, I had thought this was happening when a page was moved while I was editing it. I started forming that theory when it happened to be on a draft which was moved into mainspace while I was editing it. But, I'm reasonably sure that was just a coincidence, as per my sandbox example.

I'm also pretty confident that the edit I ran into this bug with didn't involve an edit conflict or page move. Nor do I have the Two column edit conflict beta feature enabled.

I've never managed to reproduce this before trying with edit conflicts, and trying that gave me about 50% success rate, which is much better than 0% previously. Perhaps the page just needs to have been edited recently, rather than edited recently in a conflicting manner.

RoySmith added a comment.EditedSep 18 2019, 2:31 AM

Take a look at my sandbox history: https://en.wikipedia.org/w/index.php?title=User:RoySmith/sandbox&action=history. In that run of 10 edits on 15 September, it happened twice.

PS: just to be more precise, the two times it happened are actually not shown in the history, because they didn't result in any changes being saved.

@matmarex - That could be the case. The article I edited had been edited 5 hours previously (just to set a baseline).

@mobrovac - Any ideas why RESTBase might be sending a 404 to VE in these cases?

I tried editing the itwiki page (switching editors, saving with VE, ...) but was not able to reproduce the error.

I then went to dig a bit deeper and managed to find the deeper issue, but that's still not the root cause. The reason why RESTBase returns a 404 in these cases is that the If-Match header sent by VE is a malformed UUIDv4 (i.e. TimeUUID) ID. For example, I managed to find the logs for @kaldari's example in T230272#5493542 . There, RESTBase received If-Match: 915473825/f9cb9f80-d62b-11e9-8729-85d04a2e5aa3-df; while the revision is correct, the TimeUUID is not (a valid example is 915473825/5ce59c60-da19-11e9-8134-c5c36440951a). Notice the trailing -df. I do not know why/how/when that happens, though...

Some more data. There are 35k instances of this error for the last 30 days. Around 2500 of those happened sporadically until 2019-09-02, but starting from 2019-09-03T00:00 this has started happening regularly with at least 2000 occurrences per day. Interestingly, in each instance the problem is the same: -df seems to be appended to the If-Match header. For context, this always happens when VE calls RESTBase internally (i.e., it's a VE-PHP call to RESTBase inside production) for the route /{domain}/v1/transform/html/to/wikitext/{title}/{revision}.

kaldari raised the priority of this task from High to Unbreak Now!.Sep 18 2019, 5:18 PM

Thanks for digging into this @mobrovac. Marking Unbreak Now, since this is potentially losing ~2,000 edits per day. (Once you hit this error, there is no way to save your edit without copying it into a new window.)

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptSep 18 2019, 5:18 PM

I don't know where the -df is coming from, but I found an article saying it indicates a compressed resource ("df" stands for "deflate"): https://www.thepolyglotdeveloper.com/2019/09/test-etag-browser-caching-curl-requests/.

The html parameter that was sent to the API for my edit request began with rawdeflate, in case that's a clue. Not sure if that's always the case or was an anomaly.

I reviewed VE's code and I'm pretty sure that we don't do anything interesting with etags. We just receive them from RESTBase when getting the page HTML (client-side), pass them around, and send them back to RESTBase when converting HTML to wikitext (server-side).

In fact, we don't even parse the etag, and in particular don't remove the "weak" prefix (W/), and I think that ends up included in If-Match header we send. I'm not sure if this is correct?

@kaldari I think that's a different thing. The VE client-side request to action=visualeditoredit API sends the page HTML in a compressed format, but the API code decompresses it before sending it on to RESTBase.

I reviewed VE's code and I'm pretty sure that we don't do anything interesting with etags. We just receive them from RESTBase when getting the page HTML (client-side), pass them around, and send them back to RESTBase when converting HTML to wikitext (server-side).
In fact, we don't even parse the etag, and in particular don't remove the "weak" prefix (W/), and I think that ends up included in If-Match header we send. I'm not sure if this is correct?

Above I quoted verbatim what RB receives, so at some point it does seem like something manipulates the ETag.

@kaldari I think that's a different thing. The VE client-side request to action=visualeditoredit API sends the page HTML in a compressed format, but the API code decompresses it before sending it on to RESTBase.

Since that is the case, is it possible that, because of the compression, the header gets modified automagically by either a library or the browser itself?

@kaldari I think that's a different thing. The VE client-side request to action=visualeditoredit API sends the page HTML in a compressed format, but the API code decompresses it before sending it on to RESTBase.

Since that is the case, is it possible that, because of the compression, the header gets modified automagically by either a library or the browser itself?

It's not compression on the level of the HTTP protocol though, we compress the data before we put it in a query parameter and before those are encoded into the request body. (I think there's no way to do normal HTTP compression on requests sent by the browser.)

Trizek-WMF added a subscriber: Trizek-WMF.

Mentioned in SAL (#wikimedia-operations) [2019-09-19T14:16:59Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@44f4c79]: Remove the TID suffix in the ETag, if present - T230272

Mentioned in SAL (#wikimedia-operations) [2019-09-19T14:28:19Z] <mobrovac@deploy1001> deploy aborted: Remove the TID suffix in the ETag, if present - T230272 (duration: 11m 20s)

mobrovac closed this task as Resolved.Sep 19 2019, 3:16 PM
mobrovac claimed this task.

I have deployed a temporary work-around for RESTBase that strips TID suffixes from ETags, and as of that deploy things seem to have stabilised, so the imminent problem of edits failing is now mitigated. However, this is just a work-around and we should find the proper root cause and fix the underlying problem. To that end, I have created T233320: VisualEditor <-> RESTBase communication and ETags, so let's continue the investigation there.

@Whatamidoing-WMF - Can you communicate to the communities that were discussing this bug (at least Italian and English Wikipedia) that it has been fixed (more or less)? Thanks!

Elitre removed a subscriber: Elitre.