Quoting Glaisher from T141693:

Multiple reports at enwiki and OTRS.

• https://en.wikipedia.org/wiki/Wikipedia:Help_desk#Why_No_Text_In_Article.3F
• https://en.wikipedia.org/w/index.php?diff=732245583&oldid=732245280#Unregistered_users_see_old_revisions
• https://en.wikipedia.org/w/index.php?diff=732245583&oldid=732245280#Why_is_this_article_showing_an_old_version_when_I.27m_logged_out.3F

I checked some random pages on RC while logged out and it's sometimes showing revisions which are outdated by several revisions. For example: https://en.wikipedia.org/wiki/List_of_equestrian_statues has three edits so far today but the last modified at the footer shows 29 June 2016, at 18:20 currently. Manual purging seems to fix these. I don't think this is a frequent occurrence so this is probably worth further investigation as it may be related to a recent change.

(Wondering if belated syncing of ProofRead status on Wikisource reported in T141692 might be related to caching issues.)

In T141687#2508563, @Aklapper wrote:

(Wondering if belated syncing of ProofRead status on Wikisource reported in T141692 might be related to caching issues.)

It showed me page 211 was yellow, but after purging it turned to green. Since this was while logged-in, it is probably not related to this bug as logged-in users are not supposed to be served from this cache.

Several reports have come in at the English Wikipedia's Teahouse regarding this issue. Perhaps some of their questions there can help narrow down the cause.

As my copied comment above says, the only workaround seems to be purging the page.

There haven't been any related changes recently on the cache side of things (e.g. changes in relevant VCL or how purging works, etc). More likely it's a change on the other side of the purge queue (related to... jobqueue? eventbus?). Do we have an idea when the problem started up, to correlate against changes?

In T141687#2508947, @BBlack wrote:

There haven't been any related changes recently on the cache side of things (e.g. changes in relevant VCL or how purging works, etc). More likely it's a change on the other side of the purge queue (related to... jobqueue? eventbus?). Do we have an idea when the problem started up, to correlate against changes?

The first question in the Teahouse related to this problem came in yesterday (July 29th) at about 2:16 pm (UTC-4), according to the autosigned-signature. This means that whatever happened was before that time.

The question can be found here. (If it gets archived, search for "Revised Page Not Truly Saved" in the Teahouse archives.)

Also, I was talking to someone in IRC (specifically the wikipedia-en channel), and they seemed to remember something similar from a couple of years ago. Specifically, they said:

This happened a couple years ago too I think
If I remember right, there was something that was causing IP's to go to an archived copy of the pedia that was used as a backup in case of crashes

Perhaps it's a repeat of that issue? If anyone can find the relevant ticket number for that bug, that could be helpful.

I'm not sure it is related but on Commons are also some cache problems (since 2 days) reported (unrelated logged in or not).

In T141687#2508997, @Perhelion wrote:

I'm not sure it is related but on Commons are also some cache problems (since 2 days) reported (unrelated logged in or not).

Are you saying that the caching problem on Commons is happening whether you're logged in or not?
Also, specifically, when did you start noticing the problem?

I faced this too, logged in or not, after editing https://he.wikipedia.org/wiki/%D7%A4%D7%95%D7%A8%D7%98%D7%9C:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9E%D7%95%D7%9E%D7%9C%D7%A6%D7%99%D7%9D/%D7%A2%D7%A8%D7%9B%D7%99%D7%9D/%D7%A7%D7%AA%D7%93%D7%A8%D7%9C%D7%AA_%D7%A9%D7%90%D7%A8%D7%98%D7%A8

only manual purge fixed it.

I just briefly looked through the 1.28.0-wmf.12 change log, only thing obviously from this area of core functionality was https://gerrit.wikimedia.org/r/#/c/299859/1 but it seems correct...

Probably related: I have twice (now and one or two days ago) seen version histories that lacked the most recent revision(s) when viewed as logged-out user, but were up to date when viewed while logged in. Example: https://archive.is/RLxjO . After purging the page's cache, the missing revisions showed up in the logged-out view, too.

I noticed this two days ago while visiting a user talk page on the Italian Wikipedia, where a message posted over 10 minutes before was not visible.

In T141687#2509250, @Glaisher wrote:

This issue seems more widespread than initially thought and apparently it's not just enwiki so raising priority to UBN. A user also just now reported on IRC that they were getting shown a revision containing vandalism which has been removed since then. This bug is likely causing already-reverted vandalism to be shown more frequently to the public.

I think this is a dubious argument if it's the basis of raising it to UBN. It's just as likely it's suppressing a vandalous update from public visibility as well.

I've dug through live PURGE logs to see if I can notice any odd patterns or malformed requests, but so far everything looks relatively-normal. I see the usual standard pattern where article purges come in sets of 4x (content page + ?action=history, desktop and mobile for each), and then are repeated ~12-ish seconds later to cover races. The overall rate incoming to varnish hasn't changed in any remarkable way recently, either, but I'm pretty sure there's one or more forms of ratelimiting at the jobqueue which has been hitting its cap for a long time now. Is it possible the purge jobqueue is backlogging and these edits will eventually be purged?

One pattern change (vs long term history) is there are larger bursts of REST purges happening more-frequently, but I don't know if that's new enough to be at issue here (I haven't looked at the raw log in quite some time). If that were the culprit, it would just be contributing to backlog or contending with traditional purges when it comes to packet loss in UDP network buffers, perhaps.

In general, we know current PURGE levels have been excessive to the point of dysfunction for months ( T133821 ), but things have been tending to mostly-work in spite of that, at least up until this report.

Also, can someone better explain what's going on in https://gerrit.wikimedia.org/r/#/c/299859/1/includes/page/WikiPage.php (same commit referenced earlier)? Is it possible this changed the relative timing of parser and varnish caches, such that varnish purge might pick up not-yet-invalidated parsercache output when it refreshes from its purge?

In T141687#2509530, @BBlack wrote:

varnish purge might pick up not-yet-invalidated parsercache output when it refreshes from its purge?

T140558 is one example where the varnish cache was purged after an edit, and the underlying parser cache was also more recent of the edit, but it still had stale content; that was before the changeset you mention.

In T141687#2509530, @BBlack wrote:

Also, can someone better explain what's going on in https://gerrit.wikimedia.org/r/#/c/299859/1/includes/page/WikiPage.php (same commit referenced earlier)? Is it possible this changed the relative timing of parser and varnish caches, such that varnish purge might pick up not-yet-invalidated parsercache output when it refreshes from its purge?

So, my understanding is: It used to be that WikiPage::doPurge would have Title::invalidateCache called upon the DB master connection transaction idling... But if you look at what that function does, it actually sets up code to be run on the same event (DB master connection transaction idling). So WikiPage::doPurge was changed to just call Title::invalidateCache directly (no extra TrxIdleCallback).
At a brief glance, the commit looks okay (don't assign much value to this though - I'm not confident enough in that area of the code to be properly reviewing it, it's not something I know the details of), it's just the only thing I could find on the most recently deployed wmf branch change log obviously in the area of cache purging. There are things I haven't checked in detail, for example there could be some strange issue with the state of the Title object, there could be some problem involving timing of when TrxIdleCallbacks get run that may have been bypassed before by waiting twice (?) before.
These are things someone else should review for, the problem may turn out to be something completely different and unrelated.

In T141687#2509511, @BBlack wrote:

I think this is a dubious argument if it's the basis of raising it to UBN. It's just as likely it's suppressing a vandalous update from public visibility as well.

Edits that are suppressed later on are generally reverted quite fast so they are not very public from then onwards so I don't think that can be compared. Feel free to change the priority if you think it's not a UBN though.

In T141687#2509530, @BBlack wrote:

Also, can someone better explain what's going on in https://gerrit.wikimedia.org/r/#/c/299859/1/includes/page/WikiPage.php (same commit referenced earlier)? Is it possible this changed the relative timing of parser and varnish caches, such that varnish purge might pick up not-yet-invalidated parsercache output when it refreshes from its purge?

I don't think that change be related as doPurge() is called when ?action=purge URLs are hit, not during an edit. However, there is a https://gerrit.wikimedia.org/r/#/c/299857/ which changes the commit order to be slightly different but I don't know whether that can cause this huge of a change?

I think https://gerrit.wikimedia.org/r/#/c/300338/3/includes/page/WikiPage.php is what happened here. If that is true, purges done after an edit is not executed. That means if we are seeing latest revision after an edit for logged-users, it is by another purge, not the regular purge after the edit.

Change 302117 had a related patch set uploaded (by Glaisher):
Make WikiPage::doEditUpdates and related functions be run PRESEND again

https://gerrit.wikimedia.org/r/302117

Change 302118 had a related patch set uploaded (by Aaron Schulz):
Make WikiPage::doEditUpdates and related functions be run PRESEND again

https://gerrit.wikimedia.org/r/302118

Change 302117 merged by jenkins-bot:
Make WikiPage::doEditUpdates and related functions be run PRESEND again

https://gerrit.wikimedia.org/r/302117

Change 302118 merged by jenkins-bot:
Make WikiPage::doEditUpdates and related functions be run PRESEND again

https://gerrit.wikimedia.org/r/302118

Change 302122 had a related patch set uploaded (by Aaron Schulz):
Use PRESEND for WikiPage::doCreate updates

https://gerrit.wikimedia.org/r/302122

Change 302123 had a related patch set uploaded (by Aaron Schulz):
Use PRESEND for WikiPage::doCreate updates

https://gerrit.wikimedia.org/r/302123

Change 302123 merged by jenkins-bot:
Use PRESEND for WikiPage::doCreate updates

https://gerrit.wikimedia.org/r/302123

I suspected missing PRESEND looking at this before, bug @Glaiser beat me to it.

Change 302122 merged by jenkins-bot:
Use PRESEND for WikiPage::doCreate updates

https://gerrit.wikimedia.org/r/302122

In T141687#2509033, @Gestrid wrote:

In T141687#2508997, @Perhelion wrote:

I'm not sure it is related but on Commons are also some cache problems (since 2 days) reported (unrelated logged in or not).

Are you saying that the caching problem on Commons is happening whether you're logged in or not?
Also, specifically, when did you start noticing the problem?

Not sure if that is the same issue but I heard claims of thumbnails not updating without manual purge when a new image version is uploaded. That started 2 days ago. IIRC thumbnails are cached the same way for logged-in and anonymous users (no going around varnish even if you are logged in) so it would make sense that login status does not matter.

Change 302204 had a related patch set uploaded (by Aaron Schulz):
Make recordUpload2() deferred updates PRESEND again

https://gerrit.wikimedia.org/r/302204

Change 302205 had a related patch set uploaded (by Aaron Schulz):
Make recordUpload2() deferred updates PRESEND again

https://gerrit.wikimedia.org/r/302205

Change 302205 merged by jenkins-bot:
Make recordUpload2() deferred updates PRESEND again

https://gerrit.wikimedia.org/r/302205

Thumbail deferred updates were also post-send for the last few days by oversight, which was fixed/deployed in the above patches.

Change 302204 merged by jenkins-bot:
Make recordUpload2() deferred updates PRESEND again

https://gerrit.wikimedia.org/r/302204