This seems to be completely broken as I can't save anything. It also makes a test fail:

❯ cargo test -- tests::api::test_immutable 
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.08s
     Running unittests src/lib.rs (/Users/hans/dev/countcount/mwbot/target/debug/deps/parsoid-2eaed1445b9c5a61)

running 1 test
test tests::api::test_immutable ... FAILED

failures:

---- tests::api::test_immutable stdout ----
Error: Http(reqwest::Error { kind: Status(412), url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("en.wikipedia.org")), port: None, path: "/w/rest.php/v1/transform/html/to/wikitext/Main%20Page/1225315602", query: None, fragment: None } })


failures:
    tests::api::test_immutable

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 51 filtered out; finished in 2.14s

As far as I can tell, this is working as designed: if-match checks whether a rendering with that hash is available in the parser cache or stash. If its not, the request fails woth a 412.

For pages that are rarely visited, it is likely that the parsoid rendering of the content is not in the parser cache. API requests will not cause it to be written to the parser cache either (because we don't want crawlers to cause cache eviction). So if you use if-modified with a "view" etag, it is likely to fail.

When fetching HTML content for editing, use stash=true. That should return an ETag that can then be used with if-modified. I just tested it, it seems to work. Conversion of HTML that was fetched without stashing is unreliable, and always was. RESTbase was just more agressive about practive caching.

Note that HTML fetched from the REST API without stash=true (or flavor=edit) may also be missing inline annotations needed for editing and round-trip conversion.

You are right that the documentation is lacking. We are currently working on exposing OpenAPI specs for these APIs, so their documentation can be maintained and explored more easily.

In T354037#10295304, @daniel wrote:

When fetching HTML content for editing, use stash=true. That should return an ETag that can then be used with if-modified. I just tested it, it seems to work. Conversion of HTML that was fetched without stashing is unreliable, and always was. RESTbase was just more agressive about practive caching.

Is it OK to just always set stash=true? Bots generally don't know whether they're going to make an edit until after processing the page's contents.

In T354037#10296752, @Legoktm wrote:

Is it OK to just always set stash=true? Bots generally don't know whether they're going to make an edit until after processing the page's contents.

That really depends on how often "always" is. The current rate limit is 5 per minute. I'd say that a bot that rarely edits should re-fetch the content with stash=true before editing. A bot that usually edits can just always set stash=true.

A limit of 5 edits per minute is incredibly slow (considering RESTBase let us run at 200 pages per second).

In T354037#10297755, @daniel wrote:

I'd say that a bot that rarely edits should re-fetch the content with stash=true before editing. A bot that usually edits can just always set stash=true.

This is a non-trivial rearchitecture of most bots. I honestly didn't realize the rest.php stuff was going to be such a big change otherwise I would've paid better attention to chime in earlier.

In T354037#10297918, @Legoktm wrote:

This is a non-trivial rearchitecture of most bots. I honestly didn't realize the rest.php stuff was going to be such a big change otherwise I would've paid better attention to chime in earlier.

It's not really a change in the contract though. Using if-match without stash=true was never reliable, just less likely to fail.

Avoiding pre-generation load was explicitly a design goal. That of course means that a given rendering is less likely to be available across requests, unless you request stashing.

The decision to not write to the parser cache on API requests was made later, based on observations about access patterns - the distribution curve for APIs is much more flat than for page views, so cache writes are much less likely to be useful for future reads. We have been thinking about allowing clients to control whether their request should contribute to the cache or not, but decided against it, since clients would often not know which behavior would be "good" from the server's perspective. @Krinkle and @cscott would be the people to discuss this with.

Ping @HCoplin-WMF - there's a product level decision to be made here, a trade-off between resource efficiency and the usefullness of HTML endpoints for HTML-based editing. As far as I'm aware, that use case is still relatively rare, but I think its somethign that we'd want to encourage for the future. To do that, we may need to increase the stash capacity.

There's an interesting wrinkle here - the "danger" of using the stash option too much is that the stash would overflow, which would cause old entries to get evicted, which would cause pending edits (open VE sessions) to fail. The requried capacity is a function of the write rate and the expiry time - the expiry time is currently 24h, which is the time we support for a VE session to stay open - if you keep the editor open for longer than that, you may nott be able to save your edit.

To protect the stash capacity, we have a rate limit of 5 stashes per minute. However, 24h is unnecessarily long for bots. We could probably lower it to, say, 5 minutes or even less, in exchange for a higher rate limit. Clients could perhaps specify that as a value to the stash parameter - instead of saying stash=true, they would give stash=5m, and be allowed to do that much more often than would be allowed for a 24h stash.

Another possibility would be to allow clients to explicity un-stash when they are done. That would free up capacity, but I'm not sure how to account for that in rate limiting - un-stash would have to decrement the bucket count. Not sure we support that.

All this is probabyl for later. As far as I know, Kunal's bot is the only one that would currentlly need that. Other bots generally manipulate wikitext directly. But in the long term, we'll probably want to discourage that.

I am closing this as invalid, since the API is working as designed. I filed T380015: Content API (proposal): Allow bots to opt for a lower stash ttl in exchange for a more permissive rate limit instead for discussing improvements around the stahs TTL and rate limit.

@Legoktm There may be no need for stashing or etags at all if you use the output of https://en.wikipedia.org/w/rest.php/v1/page/Earth/html?flavor=edit. flavor=edit makes it embed all the parsoid annotations in the HTML, which should allow the transform endpoint to turn HTML back into wikitext without the need for stashed state.

In T354037#10397337, @daniel wrote:

@Legoktm There may be no need for stashing or etags at all if you use the output of https://en.wikipedia.org/w/rest.php/v1/page/Earth/html?flavor=edit. flavor=edit makes it embed all the parsoid annotations in the HTML, which should allow the transform endpoint to turn HTML back into wikitext without the need for stashed state.

I'll give this a try, thanks, that would be ideal. Is this documented somewhere? I only found https://www.mediawiki.org/wiki/Parsoid/Parser_Unification/Output_flavors which seems to be about MediaWiki internals and not API options.

In T354037#10460918, @Legoktm wrote:

Is this documented somewhere? I only found https://www.mediawiki.org/wiki/Parsoid/Parser_Unification/Output_flavors which seems to be about MediaWiki internals and not API options.

Apparently not... I'll look into that. The internal flavors are similar to the ones supprted by the API though.

EDIT: I added some basic documentation to https://www.mediawiki.org/wiki/API:REST_API/Reference#Parameters_7

On my own MediaWiki instance (MediaWiki 1.43.3), today I've experience this issue. After ~30 minutes of editing a page with the visual editor, I received the "No stashed content found for" error. View changes (diff), saving the page, switching to source edit, even after adding more changes, always results in the same error. The only thing that works is the visual diff mode. I could only recover by opening the edit on a new tab, and copy&pasting the text on it. There was no conflict neither, since there were no more edits to the page.

Is there no possibility to recover from it? I'd expect it to auto-recover, just like when you get a "session lost" error when saving, but when retrying it saves correctly. The error message doesn't mean anything, and normal users may not even try to open the edit on a new tab to try the edit again. Even a big edit involving templates may not be possible to copy&paste on a new tab.