Many of the errors mention Special:Undelete (on enwiki), where a RevisionArchiveRecord is passed to our code, perhaps something in that code path is wrong.

Others are just page saves and jobs for various pages on various wikis, with no pattern that I can see.

In rEVEDf2df5dc7b98a: Improve error messages for RESTBase errors I added more details to these errors, we have three different kinds so far:

Error contacting the Parsoid/RESTBase server (HTTP 404): Page was deleted
Error contacting the Parsoid/RESTBase server (HTTP 404): Requested resource is not found.
Error contacting the Parsoid/RESTBase server (HTTP 404): Requested page does not exist.

I'll read some code to figure out what they really mean, and then either reach out to Content-Transform-Team with a more specific problem, or (if it looks beyond repair) add ParsoidOutputAccess as a fallback to RESTBase and see if that helps. I was going to use ParsoidOutputAccess for this code in the first place, before learning that it has some unspecified caching deficiencies and is not suitable for production use, but this only occurs ~50 times per day, so caching anything shouldn't be needed.

Error contacting the Parsoid/RESTBase server (HTTP 404): Page was deleted

I think this error is wrong, and the page was not in fact deleted. RESTBase will return this when the REST API [1] reports that the page exists, and the action API [2] reports that it doesn't. This can also occur when the action API reads from a stale replica, and that's probably what is happening. I stopped short of trying to figure out why it's calling two APIs.

[1] https://en.wikipedia.org/api/rest_v1/page/title/The_Fighting_Temeraire
[2] https://en.wikipedia.org/w/api.php?format=json&action=query&prop=info%7Crevisions&continue=&rvprop=ids%7Ctimestamp%7Cuser%7Cuserid%7Csize%7Csha1%7Ccomment%7Ctags&titles=The%20Fighting%20Temeraire

Ref: https://github.com/wikimedia/restbase/blob/ecef17bda6f4efc0d6e187fb05b1eeb389bf7120/lib/mwUtil.js#L224

Error contacting the Parsoid/RESTBase server (HTTP 404): Requested resource is not found.

RESTBase will return this when the action API reports that the page doesn't exist. Probably reading from a stale replica.

Ref: https://github.com/wikimedia/restbase/blob/ecef17bda6f4efc0d6e187fb05b1eeb389bf7120/sys/action.js#L148

Error contacting the Parsoid/RESTBase server (HTTP 404): Requested page does not exist.

RESTBase will return this when… its normalized title is not exactly the same as MediaWiki normalized title? I'm as surprised as you are. The error message makes no sense, but I guess we should add logging to find out what those titles are, this could be a problem for other tools…

Ref: https://github.com/wikimedia/restbase/blob/ecef17bda6f4efc0d6e187fb05b1eeb389bf7120/sys/page_revisions.js#L396

Change 829255 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] Log page/revision IDs when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/829255

Change 829255 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Log page/revision IDs when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/829255

Change 843550 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.40.0-wmf.5] Log page/revision IDs when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/843550

Change 843550 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.40.0-wmf.5] Log page/revision IDs when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/843550

Mentioned in SAL (#wikimedia-operations) [2022-10-17T20:13:36Z] <urbanecm@deploy1002> Started scap: 6762292a4: e320d48c8: 6762292a4: DicsussionTools/WikimediaEvents backports (T315688, T315689, T320938)

Mentioned in SAL (#wikimedia-operations) [2022-10-17T20:18:11Z] <urbanecm@deploy1002> Finished scap: 6762292a4: e320d48c8: 6762292a4: DicsussionTools/WikimediaEvents backports (T315688, T315689, T320938) (duration: 04m 35s)

A few errors have been recorded by now with more detailed logging, here are the results:

Reviewing this list:

• Almost all occurences are obviously bogus, as the revision and page exist. It seems that all of them have been recently created, moved or undeleted. This is consistent with my guess that something is just reading stale data.
• The 4 revisions that don't exist have been (in 3 cases) undeleted and then deleted again, or (in 1 case) deleted while reverting a page move. They have existed at the time of the error.
• The "Requested page does not exist" errors, which I thought are caused by inconsistent title normalization, occurred on pages that were recently moved, so it looks like this was also something reading stale data and comparing two different titles.
• There's one title on the list in the main namespace ("For What It's Worth"), because the page was subjected to a (reverted) page move to the talk namespace.

I think the solution is to just retry when the errors occurs, without RESTBase, like in T315689. The issue only occurs a couple dozen times per day, so this should be okay to do.

Change 844574 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] Retry without RESTBase when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/844574

Change 844574 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] Retry without RESTBase when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/844574

(I want to backport this)

Change 848391 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.40.0-wmf.6] Retry without RESTBase when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/848391

Change 848391 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.40.0-wmf.6] Retry without RESTBase when the page/revision seems to be missing

https://gerrit.wikimedia.org/r/848391

Mentioned in SAL (#wikimedia-operations) [2022-10-24T20:21:05Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:848390|Allow 'nofollow' on external links in Parsoid output (T321437)]], [[gerrit:848391|Retry without RESTBase when the page/revision seems to be missing (T315688)]]

Mentioned in SAL (#wikimedia-operations) [2022-10-24T20:21:24Z] <urbanecm@deploy1002> urbanecm and matmarex: Backport for [[gerrit:848390|Allow 'nofollow' on external links in Parsoid output (T321437)]], [[gerrit:848391|Retry without RESTBase when the page/revision seems to be missing (T315688)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet

Mentioned in SAL (#wikimedia-operations) [2022-10-24T20:27:44Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:848390|Allow 'nofollow' on external links in Parsoid output (T321437)]], [[gerrit:848391|Retry without RESTBase when the page/revision seems to be missing (T315688)]] (duration: 06m 38s)

Looks resolved indeed: