I think we saw this when it landed on beta too.
Didn't track it down then though

I am suspect that this was caused by enabling the OLD style of federation on commons in https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/485082/
before then enabling the entity source based federation https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/498165/
Looking through the code path for the new entity source based stuff I can not see why this would be happening (but open to more eyes)

The diff cache key is:

<!-- diff cache key testcommonswiki:diff:wikidiff2:1.12:old-718:rev-722:1.7.3:25:lang-en -->

DifferenceEngine in core caches diffs, which is i thought why the problem remained visible.

I went and deleted and then checked the cache key, the key correctly holds the diff now, but the page is still showing the old diff?

addshore@deploy1001:~$ mwscript eval.php --wiki testcommonswiki
> $c = \MediaWiki\MediaWikiServices::getInstance()->getMainWANObjectCache();

> $r = $c->get( "testcommonswiki:diff:wikidiff2:1.12:old-718:rev-722:1.7.3:25:lang-en" );

> var_dump($r);
string(1075) "<tr class="mw-diff-slot-header" lang="en"><th colspan="4">mediainfo</th></tr><tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">Property / <a title="d:Special:EntityPage/P180" href="https://www.wikidata.org/wiki/Special:EntityPage/P180">depicts</a></td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span><a title="d:Special:EntityPage/Q147" href="https://www.wikidata.org/wiki/Special:EntityPage/Q147">kitten</a></span></ins></div></td></tr><tr><td colspan="2" class="diff-lineno"></td><td colspan="2" class="diff-lineno">Property / <a title="d:Special:EntityPage/P180" href="https://www.wikidata.org/wiki/Special:EntityPage/P180">depicts</a>: <a title="d:Special:EntityPage/Q147" href="https://www.wikidata.org/wiki/Special:EntityPage/Q147">kitten</a> / rank</td></tr><tr><td colspan="2">&nbsp;</td><td class="diff-marker">+</td><td class="diff-addedline"><div><ins class="diffchange diffchange-inline"><span>Normal rank</span></ins></div></td></tr>"

The cache key correctly says kitten etc and does not say deleted property or item.
The page however still does, https://test-commons.wikimedia.org/w/index.php?title=File:Godward_Idleness_1900-dupe!.jpg&diff=722&oldid=718

Did I miss another layer of caching here?

Wait, i think I was hitting an in browser cache....

Yes, I was getting 304s without realizing it.

So I dont think this should happen when we move to real commons as long as the entitysource based federation is used straight out of the door.

We could verify this by rolling back federation on test commons, and then re deploying it perhaps?

But I will leave that for you to decide @Jdforrester-WMF

Update: I tried disabling "old style" federation, and it immediately and totally broke (PHP fatal, "No entity source provides properties!").

However, the problem also seems to have fixed itself (l10n cache or whatever) anyway:

Declare this fixed? Eh.

In T218921#5055949, @Jdforrester-WMF wrote:

Update: I tried disabling "old style" federation, and it immediately and totally broke (PHP fatal, "No entity source provides properties!").

Mind pointing me to the patch that was disabling it and cause the explosion? This is quite embarrassing and I'd like to have a look on what cases are not handled properly when I have a spare minute of time

In T218921#5057104, @WMDE-leszek wrote:

In T218921#5055949, @Jdforrester-WMF wrote:

Update: I tried disabling "old style" federation, and it immediately and totally broke (PHP fatal, "No entity source provides properties!").

Mind pointing me to the patch that was disabling it and cause the explosion? This is quite embarrassing and I'd like to have a look on what cases are not handled properly when I have a spare minute of time

Absolutely. "New style" is setting wmgUseEntitySourceBasedFederation to true, "old style" is setting wmgWikibaseRepoForeignRepositories with all its arrays and so on. Removal as https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/498432.