Thanks for adding this @jberkel - we'll look into it!

Just a thought: perhaps the HTML dumps should be generated from the XML dumps, so that the revisions in both match (and they can both be used interchangeably without consistency problems).

I thin this might be related to T274359.

Any updates on this? The task has been moved around a bit recently, but it's not clear what is happening. Is it difficult to fix?

Another cache fail related ticket, probably not related though: T226931

Hey, sorry for not providing updates, we are updating our infrastructure and code at the moment that is not directly related to this problem, but potentially will fix this. After we are done with the current updates, we'll double check if this is still the case, and if it is will take a closer look into it.

Thanks, are you referring to the deprecation of restbase/MCS? On the English Wiktionary, we're relying more and more on these dumps for statistics and maintenance tasks, and many editors have noticed problems with data derived from these dumps.

I crosschecked the HTML dump data with the XML dump (20230320). Around 2200 entries are missing completely from the HTML dump, here's a sample:

Additionally, around 3200 entries are older than reported in the XML dump, in extreme cases with a 12-year lag!

The last problem concerns entries that have a current timestamp, but where the content itself is outdated. This seems to happen when transcluded templates are updated after the last edit of the transcluding page. I don't have numbers for this one, but it likely affects a lot of pages.

Thanks for the stats, I'm not referring to restbase/MCS deprecation, we are doing some work WME infrastructure.

The stats themselves are really useful, would be nice to see what changes after we change our infra.

Ok, let me know once you have dumps available with the new infra and I'll re-generate them.

It looks like the situation has improved with the latest dump (20230601, enwikt):

SELECT xml.title, timestamp AS xml_timestamp, date_modified AS html_timestamp, date_diff('month', date_modified, timestamp) AS diff FROM xml
  INNER JOIN html ON xml.title = html.title 
  WHERE redirect is null AND diff > 0 AND xml_timestamp < '2023-05-26'   
 ORDER BY diff DESC

Only 12 entries have different timestamps. However, there are now duplicate entries in the HTML dump:

SELECT id, title, content_hash, count(*) AS COUNT FROM html GROUP BY id, title, content_hash HAVING count > 1 ORDER BY count DESC LIMIT 20;

There are around 114K entries which are listed more than once in the dump, and 96K are exact duplicates (matching content_hash).

Here are the changes for the top entry:

SELECT id, title, date_modified, content_hash FROM html WHERE title = 'alternating current' ORDER BY date_modified DESC LIMIT 10;

There are ~150 entries missing from the HTML dump (compared to 2200 earlier):

SELECT xml.title, timestamp FROM xml LEFT OUTER JOIN html ON xml.title = html.title WHERE xml.redirect IS NULL AND html.title IS NULL AND timestamp < '2023-05-25' LIMIT 10;

Conversely, around 200 entries are in the HTML dump, but not in the XML dump: in most cases these are entries that have been deleted.

Why was this already marked as resolved? New dumps haven't even been published yet, so it's impossible to verify.

@jberkel Thank you for the updated info. The recent changes have not made it into our production infra yet, they are still in staging. I will keep you posted on when they do.

Ok, I hope this can be rolled out quickly, it can't get much worse than the current state

investigation timeboxed for 5 days and made by @LDlulisa-WMF was completed. With the results of the investigation, We´ll triage accordingly during the next sprint planning.

BTW, there is a related issue T300124 that some categories and templates are not always extracted.

file sizes from the most recent enwikt HTML dumps (NS0):

20230701: 13 GB
20230720: 7.1 GB
20230801: 1.1 GB
20230820: 4.6 GB
20230901: 7.2 GB
20230920: 3 GB
20231001: 5 GB

something's going really wrong there.

Every MediaWiki site publishes a count of its own "content" pages at the API endpoint https://de.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics . The definition of "content" can be confirmed through the https://www.mediawiki.org/w/api.php?action=query&meta=siteinfo&siprop=namespaces API, and for most wikis this will be strictly NS0.

Please include some kind of basic health check after each dump run, to verify that the article count matches these numbers.

Sadly, the stats are not easily available for other namespaces (while dumps are made for them as well). I think currently the best way to get those counts is to use ElasticSearch endpoint, as described in T312200.

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

In T305407#9223782, @jberkel wrote:

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

When you say "base off them", are you suggesting that the HTML dumps be produced by iterating over the XML dumps and then fetching the HTML content for each row? This would be a cumbersome approach since it introduces an unnecessary extra dependency. The HTML content of the new dumps is not directly derived from the XML dumps, so I don't see much advantage to this approach. I agree that it would be nice to snapshot the wiki content before dumping but this isn't feasible given the way that HTML rendering requires random access to all articles and templates.

In T305407#9223796, @awight wrote:

In T305407#9223782, @jberkel wrote:

I've suggested this previously: the XML dumps have been around for a very long time, and compared to the HTML version, they are very reliable, so why not simply base the HTML dumps off them? Then you could easily compare the counts of the two files, it would also help users who consume both types of data.

When you say "base off them", are you suggesting that the HTML dumps be produced by iterating over the XML dumps and then fetching the HTML content for each row? This would be a cumbersome approach since it introduces an unnecessary extra dependency. The HTML content of the new dumps is not directly derived from the XML dumps, so I don't see much advantage to this approach. I agree that it would be nice to snapshot the wiki content before dumping but this isn't feasible given the way that HTML rendering requires random access to all articles and templates.

I don't know how the HTML dumps are currently produced (the project is not very transparent), but you probably don't have to transform the XML itself, you could take the page ids and revision numbers from the XML dump as a starting point, and then fetch the HTML of those revisions. Of course this is more complicated with templates dependencies, but it would already be an improvement, and ensure the data you get is complete.

so why not simply base the HTML dumps off them?

This does not work. You cannot convert wikitext to HTML in offline mode. You need access to running Mediawiki instance to render macros, templates, etc. So rendering HTML is by definition a dynamic process and something which has not been available otherwise.

In T305407#9223814, @Mitar wrote:

so why not simply base the HTML dumps off them?

This does not work. You cannot convert wikitext to HTML in offline mode. You need access to running Mediawiki instance to render macros, templates, etc. So rendering HTML is by definition a dynamic process and something which has not been available otherwise.

I'm not suggesting this, see my reply above. The XML dumps would serve as a baseline for which page ids / revisions to include.

I ran into another surprising quirk in these dumps: a page may appear multiple times, sometimes with a duplicate revision, or with several different revisions. Please deduplicate by page ID.

For example, the dewiki-20230820 NS0 dump contains these revisions of one article:

["Haus Lohe",192272812]
["Haus Lohe",236444682]
["Haus Lohe",236444682]
["Haus Lohe",236444682]
["Haus Lohe",236444682]

adding some more stale data examples I came across while working with a snapshot of the HTML dumps, specifically /public/dumps/public/other/enterprise_html/runs/20231201/enwiki-NS0-20231201-ENTERPRISE-HTML.json.tar.gz on PAWS:

• A redirect that was briefly vandalized in July 2023 to not be a redirect: https://en.wikipedia.org/w/index.php?title=Adhirath&oldid=1167382940
• This also very old briefly-an-article-now-a-redirect: https://en.wikipedia.org/w/index.php?oldid=1025016513