I just made edit there https://it.wikipedia.org/w/index.php?title=Utente:IndyJrBot/ListaListeDaSpostare&diff=107085812&oldid=107083041 without problems.

Try to made more deleting lines and modify wikilink. The problem cannot reproduce at every single edit, but it appears very frequently.

Possibly this discussion also demonstrating the issue.

Just another instance of the same thing seen by kaldari, above:

action: visualeditoredit
format: json
formatversion: 2
campaign: 
summary: c
watch: 
paction: save
errorformat: html
errorlang: en
errorsuselocal: 1
page: User:RoySmith/sandbox
oldid: 915824451
basetimestamp: 20190915142553
starttimestamp: 20190915142603
etag: "915824451/bb5d8160-d7c4-11e9-87b1-2d30288168bb-df"
html: [...]
token: c8538334105dcb91b1fe954f7a5975cd5d7e49cb \

{"errors":[{"code":"apierror-visualeditor-docserver-http","html":"HTTP 404","module":"visualeditoredit"}],"docref":"See https://en.wikipedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes.","servedby":"mw1287"}

The problem I reported still go on, on random pages, and is not reproducible. The only workaround I found to don't lose job until then, is to copy and paste into a new window that NOT recovers the unsaved work (because otherwise it doesn't save it again with the same error message).

Until now we had no idea what was causing those, but in the most recent bug report at T233127 @Waddie96 discovered that this is caused by the "Two column edit conflict" beta feature. Thank you for that! I can confirm that if you have it enabled and have an edit conflict, you get this weird error message instead of the expected one (like below).

I don't know yet why this happens, but now we at least have something we can debug.

After some more testing, I don't think it's about the two column edit conflict beta feature, but just edit conflicts in general. I also get the issue sometimes when the beta feature is disabled.

The 404 error is generated by RESTBase. When VE requests RESTBase to transform the HTML to wikitext, we need to pass along the page title, revision ID, and an "etag" (an unique ID for the parsed document). We receive that etag from RESTBase itself when requesting the page HTML for editing. When trying to transform new HTML to wikitext later, RESTBase sometimes returns the 404 error, as if the document it gave us earlier suddenly disappeared.

I don't think it's edit conflict related. I've seen this several times when there was no edit conflict. For example, the one I cited above that happened on User:RoySmith/sandbox. For sure, there was no edit conflict on my sandbox.

At one point, I had thought this was happening when a page was moved while I was editing it. I started forming that theory when it happened to be on a draft which was moved into mainspace while I was editing it. But, I'm reasonably sure that was just a coincidence, as per my sandbox example.

I'm also pretty confident that the edit I ran into this bug with didn't involve an edit conflict or page move. Nor do I have the Two column edit conflict beta feature enabled.

I've never managed to reproduce this before trying with edit conflicts, and trying that gave me about 50% success rate, which is much better than 0% previously. Perhaps the page just needs to have been edited recently, rather than edited recently in a conflicting manner.

Take a look at my sandbox history: https://en.wikipedia.org/w/index.php?title=User:RoySmith/sandbox&action=history. In that run of 10 edits on 15 September, it happened twice.

PS: just to be more precise, the two times it happened are actually not shown in the history, because they didn't result in any changes being saved.

@matmarex - That could be the case. The article I edited had been edited 5 hours previously (just to set a baseline).

@mobrovac - Any ideas why RESTBase might be sending a 404 to VE in these cases?

I tried editing the itwiki page (switching editors, saving with VE, ...) but was not able to reproduce the error.

I then went to dig a bit deeper and managed to find the deeper issue, but that's still not the root cause. The reason why RESTBase returns a 404 in these cases is that the If-Match header sent by VE is a malformed UUIDv4 (i.e. TimeUUID) ID. For example, I managed to find the logs for @kaldari's example in T230272#5493542 . There, RESTBase received If-Match: 915473825/f9cb9f80-d62b-11e9-8729-85d04a2e5aa3-df; while the revision is correct, the TimeUUID is not (a valid example is 915473825/5ce59c60-da19-11e9-8134-c5c36440951a). Notice the trailing -df. I do not know why/how/when that happens, though...

Some more data. There are 35k instances of this error for the last 30 days. Around 2500 of those happened sporadically until 2019-09-02, but starting from 2019-09-03T00:00 this has started happening regularly with at least 2000 occurrences per day. Interestingly, in each instance the problem is the same: -df seems to be appended to the If-Match header. For context, this always happens when VE calls RESTBase internally (i.e., it's a VE-PHP call to RESTBase inside production) for the route /{domain}/v1/transform/html/to/wikitext/{title}/{revision}.

I don't know where the -df is coming from, but I found an article saying it indicates a compressed resource ("df" stands for "deflate"): https://www.thepolyglotdeveloper.com/2019/09/test-etag-browser-caching-curl-requests/.

The html parameter that was sent to the API for my edit request began with rawdeflate, in case that's a clue. Not sure if that's always the case or was an anomaly.

I reviewed VE's code and I'm pretty sure that we don't do anything interesting with etags. We just receive them from RESTBase when getting the page HTML (client-side), pass them around, and send them back to RESTBase when converting HTML to wikitext (server-side).

In fact, we don't even parse the etag, and in particular don't remove the "weak" prefix (W/), and I think that ends up included in If-Match header we send. I'm not sure if this is correct?

@kaldari I think that's a different thing. The VE client-side request to action=visualeditoredit API sends the page HTML in a compressed format, but the API code decompresses it before sending it on to RESTBase.

In T230272#5504398, @matmarex wrote:

I reviewed VE's code and I'm pretty sure that we don't do anything interesting with etags. We just receive them from RESTBase when getting the page HTML (client-side), pass them around, and send them back to RESTBase when converting HTML to wikitext (server-side).

In fact, we don't even parse the etag, and in particular don't remove the "weak" prefix (W/), and I think that ends up included in If-Match header we send. I'm not sure if this is correct?

Above I quoted verbatim what RB receives, so at some point it does seem like something manipulates the ETag.

@kaldari I think that's a different thing. The VE client-side request to action=visualeditoredit API sends the page HTML in a compressed format, but the API code decompresses it before sending it on to RESTBase.

Since that is the case, is it possible that, because of the compression, the header gets modified automagically by either a library or the browser itself?

In T230272#5504464, @mobrovac wrote:

@kaldari I think that's a different thing. The VE client-side request to action=visualeditoredit API sends the page HTML in a compressed format, but the API code decompresses it before sending it on to RESTBase.

Since that is the case, is it possible that, because of the compression, the header gets modified automagically by either a library or the browser itself?

It's not compression on the level of the HTTP protocol though, we compress the data before we put it in a query parameter and before those are encoded into the request body. (I think there's no way to do normal HTTP compression on requests sent by the browser.)

Mentioned in SAL (#wikimedia-operations) [2019-09-19T14:16:59Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@44f4c79]: Remove the TID suffix in the ETag, if present - T230272

Mentioned in SAL (#wikimedia-operations) [2019-09-19T14:28:19Z] <mobrovac@deploy1001> deploy aborted: Remove the TID suffix in the ETag, if present - T230272 (duration: 11m 20s)

@Whatamidoing-WMF - Can you communicate to the communities that were discussing this bug (at least Italian and English Wikipedia) that it has been fixed (more or less)? Thanks!

Done for Italian, https://it.wikipedia.org/wiki/Wikipedia:Officina#Risoluzione_di_un_problema .