In T233178#5574532, @ArielGlenn wrote:

I have found the reason for triple lookups of content; the third lookup, performed from within the Abstract Filter extension, is to determine whether or not the revision is a redirect or not.

Thank you for investigating, Ariel!

Pinging Performance-Team since they own WANObjectCache.

I put this up for SWAT in 3 hours: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/542963

The handler for this route doesn't seem to check read permissions for the pages it is comparing. That means it could be used to bypass per-page read permissions. The DifferenceEngine class responsible for showing diffs in the UI does apply these checks, though I note that ApiComparePages seems to be lacking them as well.

On the patch @BPirkle asked:

Thank you Daniel. Was that comment informational, or did you see something we need to change?

In T234636#5558602, @kostajh wrote:

@daniel sure I'd be happy to help!

I've updated the task, making the two of you leaders.

In T235188#5566130, @Verdy_p wrote:

Is there any chance that the Memcache server has incorrect clock setting (or NTP failing its updates) ?

In T224949#5565266, @Reedy wrote:

I got bored of just kicking the failing test can down the road after 5 supporting backports to REL1_32 and REL1_33 to help this change

In T235188#5566029, @Verdy_p wrote:

Another message duplicated:
"<strong>$1</strong> a été effacé."

In T235188#5566008, @Verdy_p wrote:

There's somthing strange in your Memcache query; "rev_user_text:" is empty for many ones. Aren't revisions supposed to be associated to a user ?

I'd be very interested to know whether the code in https://gerrit.wikimedia.org/r/542325 would pass on your live system. Of course, the test doesn't use the real cache setup. So to test that, you'd have to run equivalent code in eval.php. This would introduced a handful of orphan rows into the text table each time you run it. Not great, but maybe ok for a one-off?

It seems that we have enough information to pinpoint the cause.

Should this user story mention that unauthorized users should be unable to read this data?

A quick heads up on permission checks when serving content, because I didn't see any permission checks in the patch:

A quick search tells me that the new SqlBlobStore::getBlobBatch method is the *only* code that uses WANObjectCache::getMultiWithUnionSetCallback(). So that code hasn't previously been exercised. I suspect it might be buggy. It does have decent test coverage, but still...

In T235188#5564873, @TK-999 wrote:

Interesting, I've seen a similar caching issue where revision content got mixed up on a 1.33 wiki with $wgMultiContentRevisionSchemaMigrationStage = SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW;. I haven't yet been able to consistently reproduce it, though.

Has the new version of quibble been deployed everywhere?
Should CI on the api-testing repo now be using the hardcoded secret?

For the record, I still have no clue how the cache gets corrupted, and no good idea for investigating it.
I wonder what the cache keys look like. maybe they are too long, and they get truncated somehow? That would do it.

In T235027#5563431, @Nikerabbit wrote:

In T235027#5560244, @daniel wrote:

We have seen some mystery problems that may be related... I can't find it right now, @Anomie would know.

Are the mystery problems anything like T235188: Preemptive refresh in getMultiWithSetCallback() and getMultiWithUnionSetCallback() pollutes cache?

Slightly less insane option:
$wgMainWANCache = CACHE_NONE;

@Nikerabbit a hacky way to investigate: in SqlBlobStore::getBlob, rip out the caching code. should look something like this:

In T235188#5563527, @Anomie wrote:

@Nikerabbit: Try this:

SELECT rev_text_id, slots.*, content.* FROM revision LEFT JOIN slots ON (slot_revision_id = rev_id) LEFT JOIN content ON(content_id = slot_content_id) WHERE rev_id = 7714145;

When investigating btw, beware that diffs are cached, and there is no simple way to purge that cache.
So even once the actual issue has been fixed, you will see bad diffs.

@Nikerabbit what version are you running? I merged some changes against the Language class the other day. Though I don't see how that would mess with page history.

In T235188#5563482, @Nikerabbit wrote:

I'm not sure how the slots and content play role in this, but it seems that it is not corrupted in the text table at least. Where am I supposed to see content_address?

In T235188#5563473, @Verdy_p wrote:

Anyway on the affected wikis, some users start getting insulted or banned even if they did not make the breaking changes.
Havoc starts spreading to random places, and the user's history is now definitely wrong for what they really did.

Two things to investigate:

1. kill the cache and see if that fixes things
2. look at the content_address in the database and the content in the corresponding text row.

If this has anything to do with our changes to SqlBlobStore, my primary suspect is the caching logic. getBlobBatch() uses WANObjectCache::makeMultiKeys and WANObjectCache::getMultiWithUnionSetCallback. If anything went wrong with these cache keys, that would explain your observation.

That does sound scary. But I'm not aware of recent changes to how revision content gets loaded.
We added bulk code, but the code path is nearly entirely separate. They converge in SqlBlobStore::getBlob calling SqlBlobStore::fetchBlobs(). That code landed in master on August 30, see
https://gerrit.wikimedia.org/r/c/mediawiki/core/+/532449. I'm not aware of any change to the "normal" content loading since then. We only worked on the bulk code, as far as I am aware.

In T193613#5561446, @Milimetric wrote:

I'm thinking about the generated documentation that may now be confusing to newcomers / people who haven't read this policy yet. For example, if you have two public methods, and one of them is annotated with @stable while the other is not. I would read the documentation and use any public method and expect it to be stable unless something jumped out at me telling me not to make this assumption. So, ideally, docs would color/explain both annotated methods and public methods that are not annotated.

Per the TechCom meeting on October 9, this RFC is entering the Last Call period. If there are no pertienent concerns raised and left unaddressed by October 23, this RFC will be approved as proposed.

$rev->getSlots() returns raw slots, with no audience checks. So while the parameter in getContent( RevisionRecord::RAW ) is incorrect, the code still does the right thing.

Not sure what CPT can do here. Tagging for triage.

Patch was merged, removing the patch for review tag.

Could the opposite happen as well: it would not find the latest revision, and then fetch stale content (penultimate)?

To investigate, any errors reported in the StatusValue returned by the calls to getContentBlobsForBatch() could be logged. We currently just ignore them.

In T231671#5559794, @CCicalese_WMF wrote:

@daniel, you now have the epic as a subtask of the user story. It should be the other way around. Are we missing an engineering task here?

In T230607#5559753, @CCicalese_WMF wrote:

Does this imply that the current MCR schema migration initiative is focused only on the revision table and that there will need to be a separate page table schema migration activity later?

In T233178#5557894, @CCicalese_WMF wrote:

@daniel/@Pchelolo was this task moved to the Done column in error? I see that it still has a patch set open.

In T234649#5558176, @kaldari wrote:

@Mooeypoo @daniel @Krinkle - Any of y'all want to be session leaders or facilitators for this? Feel free to ping other folks as well.

@kostajh I just volunteered to lead this. Want to be the co-lead?

In T234995#5557541, @Pchelolo wrote:

The api-testing has abstractions and some tests set up for Action API, so we would probably need to invest a bit into setting it up for REST API routes as well before we can write actual tests.

One possible concern I have for using it right away is that the REST API seem to not yet be very stable, and moving the tests into a separate repo adds a little bit of overhead when changing the API.. I think we should start off the right track and invest in supporting REST routes in api tester.

In T231588#5557356, @eprodromou wrote:

Any thoughts or concerns there?

Please just return the HTML from the default parser.