Corruption of text from early 2005 due to HistoryBlobStub pointers broken by recompressTracked.php
Closed, ResolvedPublic


In several articles, revision text from early 2005 appears blank when viewed in the English Wikipedia. IIRC the blank revision text appears around the time that Wikipedia changed its compression formats. This has been discussed several times on the English Wikipedia village pump; the URLs below contain plenty of examples of this problem:

Version: unspecified
Severity: major
URL: 65#Blank_sequence_of_pages_in_an_article's_history

bzimport set Reference to bz20757.
Graham87 created this task.Via LegacySep 21 2009, 2:32 PM
brion added a comment.Via ConduitSep 21 2009, 4:49 PM

Tim, can you take a peek?

ISTR we cleaned up some similar items recently, where the old revs had ended up stored with incorrect compression flags which lead to them being loaded incorrectly... we might have more of such. :(

Platonides added a comment.Via ConduitSep 21 2009, 5:22 PM

This may be the same issue faced by, regarding compressed revisions.

Tobias added a comment.Via ConduitSep 21 2009, 11:05 PM

Might be related:
This diff says there are 2950 intermediate revisions, but there are none in the history:

This diff doesn't say there are any intermediate revisions, however one revision is from 2006 and another one from 2004 - there are at least hundreds of intermediate revisions.

Graham87 added a comment.Via ConduitOct 27 2009, 6:43 AM

Also see this current village pump discussion:

Do you need *all* the examples of this issue to be reported here, or can a database query or some other process be used to fix all the places where this occurred, like what happend at bug 19990?

Graham87 added a comment.Via ConduitNov 8 2009, 5:49 AM

There are also a few more instances at:

in the sections entitled "Lost page histories" (described above), "Missing revision content on Magic Knight Rayearth", and "revision history oddities".

Also see this current discussion:
Under the section "Blank revisions, tracking them".

bzimport added a comment.Via ConduitNov 16 2009, 3:57 PM

asmarin wrote:

I have similar problem with my site

I use CompressOld.php over my database and has a bug when release was 1.14.x.

If you see recent changes before apply compressold dont show nothing.

On 1.14.x runs, but on > 1.15 dont. See

A patch wasnt release never.

Graham87 added a comment.Via ConduitNov 24 2009, 12:22 PM

The problem also occurs, somewhat ironically, at this revision:

Graham87 added a comment.Via ConduitDec 10 2009, 6:47 AM

Another example is here, in the edits before the cut and paste move:

I won't history merge it yet, to avoid what happened in bug 19990.

TheDJ added a comment.Via ConduitDec 16 2009, 12:33 PM

And one more:

All edits between (21:01, 11 January 2005) and (02:37, 25 April 2005) got fubared. They show up as blank versions.

Graham87 added a comment.Via ConduitDec 18 2009, 9:53 AM

All edits from 2005, besides the first and the last ones from that year, are blank in this page history:

bzimport added a comment.Via ConduitJan 22 2010, 8:23 PM

Moonriddengirl wrote:

The bug occurs in Buckingham Palace, from to

(The text briefly reappears in the middle of the range, [], but evidently just for one edit.)

It also occurs in Norway. to

We may find more instances, as we're reviewing a number of edits at

I've asked for these to be noted.

tstarling added a comment.Via ConduitFeb 8 2010, 6:49 AM

This is not what I thought it was. It is a bug in recompressTracked.php. I am looking at it now. It should be recoverable.

tstarling added a comment.Via ConduitFeb 8 2010, 7:35 AM

OK I've checked a lot of these test cases, and they all seem to be the same, so I'm changing the summary. All of the relevant revisions should now be serving errors instead of pretending to be blank.

The original version of compressOld.php concatenated several revisions into one "blob" and stored it in a random row in the old table. Then the other old rows which needed data from the concatenated blob would get a pointer object, called a HistoryBlobStub. This pointer object gave an old_id and content hash which located the text for that revision.

After we started using external storage (ES), all the bulk data was moved out of the core database. Now, to load a HistoryBlobStub, MW would first load the old_id where the concatenated text used to be, where it would find a second pointer (with old_flags=external), then it would follow the second pointer to load the blob from ES. This was an inefficient situation, so I introduced a new pointer type (the "two-part CGZ URL") which pointed directly from the rows where the stub objects used to be, into ES.

I then wrote a script called resolveStubs.php, and ran it, removing all HistoryBlobStub objects from the database. Or at least, that's what I thought I did. It transpires that these missing revisions above are all HistoryBlobStub objects that somehow escaped resolveStubs.php.

The current generation of recompression script, trackBlobs/recompressTracked, has no appropriate handling for HistoryBlobStub. It leaves the HistoryBlobStub objects in place, but removes the CGZ objects they point to, creating a broken pointer.

Due to a bug in Revision.php, the broken pointer was displayed as a blank page instead of an error message. This is fixed in r62119.

Luckily I was fairly paranoid when I wrote trackBlobs/recompressTracked, and all the data required for recovery appears to have been retained. It's just a matter of writing a bug fix script.

Graham87 added a comment.Via ConduitFeb 8 2010, 8:26 AM

Thanks Tim for looking into this. I've added some text about this bug to:

It'd be confusing to have this error message pop up when someone is checking the history of a page. Since I had to read through your explanation twice to understand, I hope that "database glitch" is OK for now as a layman's explanation.

Platonides added a comment.Via ConduitFeb 8 2010, 10:35 PM

That doesn't explain the existance of wrong ConcatenatedGzipHistoryBlob objects (the serialized mItems length doesn't match with the real one).
Perhaps they were indeed different issues :S

tstarling added a comment.Via ConduitFeb 8 2010, 10:37 PM

Report different issues on a separate bug report please.

tstarling added a comment.Via ConduitFeb 11 2010, 2:56 AM

All the test cases on the English Wikipedia should be fixed now:

  • 1.3 million revisions were broken by this bug and are now fixed
  • 177 revisions were unrecoverable due to being damaged by a previous compression script some years ago, while cluster4 and cluster5 were current.
  • 333 revisions were unrecoverable due to the text row being missing, probably due to a bug in the original 2005 compression script.

The fix script still needs to be run on the other wikis, so this bug has to stay open for now.

Dinoguy1000 added a comment.Via ConduitFeb 11 2010, 4:54 PM

Are you going to provide a list of the unrecoverable revisions?

tstarling added a comment.Via ConduitFeb 12 2010, 12:13 AM

They're not really relevant to this bug. Maybe they are listed on some other bug report already.

Graham87 added a comment.Via ConduitMar 31 2010, 2:42 PM

Does this error message at the plasma page have anything to do with bug 20,757, or the fix for it:

I undid a braindead history merge from "plasma" to "plasma physics", before the script was run in the English Wikipedia. Since the history merge tangled many edits together from January 2005, I wonder if my machinations at the plasma and plasma physics pages in January 2010 caused something to break.

I'm fairly sure that the above revision was visible before I untangled the history at plasma physics.

ArielGlenn added a comment.Via ConduitMay 28 2010, 2:40 AM

So Tim ran the fixup script on all other wikis on Feb 27th and none of them were affected. I don't know if there is anything else that needs to be checked before this bug is closed, though.

Graham87 added a comment.Via ConduitMay 28 2010, 2:58 AM

The bug is almost resolved, then. I'm still curious about the problem with the plasma article that I described in comment 29; it turns out to affect all edits from 12:22, 28 January 2005 (UTC_) to 00:00, 16 April 2005 (UTC). I'd like to know whether (a) it is a result of this bug and (b) whether the affected revisions are recoverable.

ArielGlenn added a comment.Via ConduitMay 28 2010, 6:09 AM

(In reply to comment #31)

According to the fixup script, those revisions are unrecoverable.

I had a look at a few random revisions 9752546, 11243046, 11397897 from the time period you mentioned. The text pointer for these revisions goes to a single location in cluster5, with the same id and itemid. I seem to be able retrieve something from there manually, plugging the pointer into ExternalStore::fetchFromUrl(), but it's one text item, not a concatenated set of texts. I can't say if your history unmerge had anything to do with it.

TheDJ added a comment.Via ConduitMay 28 2010, 12:34 PM

Ariel, can you check 44320111 from bug 8689 against that list ?

Perhaps the list of unrecoverable revisions be added to the ticket or something ? That would help match any other cases we find against this problem and help finding issues that are something other than this problem.

bzimport added a comment.Via ConduitJul 9 2011, 2:55 AM

EN.WP.ST47 wrote:

It seems that between Tim and Ariel the repair scripts have been run and all test cases except the most recent one referenced to bug 8689, however that bug has been resolved, and the referenced revision text appears to be available. Marking this as fixed?

Add Comment