Page MenuHomePhabricator

Data missing from en.wiktionary.org February 2026 "MediaWiki Content File Exports" compared to "XML Database dump"
Closed, ResolvedPublicBUG REPORT

Description

Comparing the old XML database dump with the new MediaWiki Content File Exports dump:

  1. there are ~10,000 mainspaces pages that exist in the XML dump but not the ContentFileExport
  2. there are 11 mainspaces pages are in the ContentFileExport but not the old dump (the pages were deleted from the wiki between early December and late January)
  3. there are 2282 pages in the ContentFileExport that are missing the page text (I think this happened before with the old dumps and was fixed sometime in the last year).
    1. Some of these seem to be getting miscategorized as redirects

Download the files

wget https://dumps.wikimedia.org/enwiktionary/20260201/enwiktionary-20260201-pages-articles.xml.bz2
wget https://dumps.wikimedia.org/other/mediawiki_content_current/enwiktionary/2026-02-01/xml/bzip2/enwiktionary-2026-02-01-p6p11451351.xml.bz2

Check the sha sums against the published sums (they match)

sha1sum enwiktionary-20260201-pages-articles.xml.bz2
9a2274a328d56b33641322abd4bde64059c67309  enwiktionary-20260201-pages-articles.xml.bz2

sha256sum enwiktionary-2026-02-01-p6p11451351.xml.bz2
9b7a0d4f8563b7e5d77315347052e4f8336a88e820f4b36e7af758a5977ffd1c  enwiktionary-2026-02-01-p6p11451351.xml.bz2

Crude method to dump mainspace titles

bzcat enwiktionary-2026-02-01-p6p11451351.xml.bz2 | grep '^    <title>' | grep -v ':' | pv -l > enwiktionary-2026-02-01-p6p11451351.xml.bz2.allpages
bzcat enwiktionary-20260201-pages-articles.xml.bz2 | grep '^    <title>' | grep -v ':' | pv -l > enwiktionary-20260201-pages-articles.xml.bz2.allpages

Count pages in each dump

wc -l *allpages
  8741706 enwiktionary-2026-02-01-p6p11451351.xml.bz2.allpages
  8751224 enwiktionary-20260201-pages-articles.xml.bz2.allpages

First 10 pages that exist in the XML dump but not the ContentFileExport (these have existed for a long time on the wiki and should be in the dump)

diff enwiktionary-2026-02-01-p6p11451351.xml.bz2.allpages enwiktionary-20260201-pages-articles.xml.bz2.allpages | grep "^>" | head

>     <title>pie</title>
>     <title>A</title>
>     <title>fable</title>
>     <title>ability</title>
>     <title>abnegation</title>
>     <title>week</title>
>     <title>servant</title>
>     <title>second</title>
>     <title>perro</title>
>     <title>because</title>

Pages that exist in the ContentFileExport but not the XML dump (all deleted in December or January)

diff enwiktionary-2026-02-01-p6p11451351.xml.bz2.allpages enwiktionary-20260201-pages-articles.xml.bz2.allpages | grep "^<"

<     <title>A/translations</title>
<     <title>白苹</title>
<     <title>tiú</title>
<     <title>aẓermaḍ</title>
<     <title>wokenese</title>
<     <title>Ericsdott</title>
<     <title>barachoises</title>
<     <title>giggleshitter</title>
<     <title>bóina</title>
<     <title>chip sandwiches</title>
<     <title>𑀘𑁂𑀬𑁆𑀯𑀼</title>

Empty pages in the ContentFileExport

bzcat enwiktionary-20260201-pages-articles.xml.bz2 | grep '^      <text bytes="[^"]*" sha1="[^"]*" xml:space="preserve"></text>' | wc -l
2282

No empty pages in the XML dump

bzcat enwiktionary-2026-02-01-p6p11451351.xml.bz2 | grep '^      <text bytes="[^"]*" sha1="[^"]*" xml:space="preserve"></text>' | wc -l
0

Sample of an empty page misclassified as a redirect in the ContentFileExport

bzcat enwiktionary-2026-02-01-p6p11451351.xml.bz2 | grep -B1 -A20 '<title>Template:wikipedia inline</title>'
  <page>
    <title>Template:wikipedia inline</title>
    <ns>10</ns>
    <id>71571</id>
    <redirect title="Template:pedia" />
    <revision>
      <id>87466915</id>
      <parentid>82601960</parentid>
      <timestamp>2025-10-17T17:56:34Z</timestamp>
      <contributor>
        <username>Juwan</username>
        <id>3726457</id>
      </contributor>
      <minor />
      <comment>Juwan moved page [[Template:pedia]] to [[Template:wikipedia inline]]: Standardize title: [[Wiktionary:Grease pit/2025/September#Cleanup of interwiki templates]]</comment>
      <origin>67951436</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="134" sha1="2b9gh7qf7g5eo9oi85oc5p8kyhobye9" xml:space="preserve"></text>
      <sha1>2b9gh7qf7g5eo9oi85oc5p8kyhobye9</sha1>
    </revision>
  </page>

Same page in the XML dump includes page content and is not a redirect

bzcat enwiktionary-20260201-pages-articles.xml.bz2 | grep -B1 -A20 '<title>Template:wikipedia inline</title>'

  <page>
    <title>Template:wikipedia inline</title>
    <ns>10</ns>
    <id>71571</id>
    <revision>
      <id>87466915</id>
      <parentid>82601960</parentid>
      <timestamp>2025-10-17T17:56:34Z</timestamp>
      <contributor>
        <username>Juwan</username>
        <id>3726457</id>
      </contributor>
      <minor />
      <comment>Juwan moved page [[Template:pedia]] to [[Template:wikipedia inline]]: Standardize title: [[Wiktionary:Grease pit/2025/September#Cleanup of interwiki templates]]</comment>
      <origin>67951436</origin>
      <model>wikitext</model>
      <format>text/x-wiki</format>
      <text bytes="134" sha1="2b9gh7qf7g5eo9oi85oc5p8kyhobye9" xml:space="preserve">{{#invoke:interproject|projectlink|prefix=w|name=Wikipedia|image=Wikipedia-logo.svg|compat=1}}&lt;noinclude&gt;{{documentation}}&lt;/noinclude&gt;</text>
      <sha1>2b9gh7qf7g5eo9oi85oc5p8kyhobye9</sha1>
    </revision>
  </page>

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
bump version mwch pipelinesrepos/data-engineering/airflow-dags!2123a-pizzataT417596-add-redirectsmain
Customize query in GitLab

Event Timeline

@JeffDoozan thank you for the detailed, reproducible bug report!


@APizzata-WMF: FYI. I suspect these may be issues with our reconciliation mechanism.

@APizzata-WMF: FYI. I suspect these may be issues with our reconciliation mechanism.

I will test the new algorithm to see if we catch this case

I will test the new algorithm to see if we catch this case

After testing it appears that the problem is due to missing events of redirects in our datalake. I have a possible fix for that under test, will communicate more once I have results and a deployment plan.

In this MR we are detecting the wrong redirect and reconcile the combination of page_id and revision_id that are not correct.

Pipeline has been merged, will wait for monthly reconcile to check the results and see if there are any other issues. @JeffDoozan please let us know if you find any other inconsistencies.

The query:

spark.sql(f"""
    select
        page_redirect_target 
    from wmf_content.mediawiki_content_history_v1 
    where 
        page_id= 71571 
        and wiki_id = 'enwiktionary' 
        and revision_id=87466915"""
).show(truncate=False)

+--------------------+
|page_redirect_target|
+--------------------+
|null                |
+--------------------+

Now returns the correct result compared to :

spark.sql(f"""
    select
        page_redirect_target 
    from wmf_content.mediawiki_content_history_v1 timestamp as of '2026-03-31 00:00:00'
    where 
        page_id= 71571 
        and wiki_id = 'enwiktionary' 
        and revision_id=87466915"""
).show(truncate=False)

+--------------------+
|page_redirect_target|
+--------------------+
|Template:pedia      |
+--------------------+

@JeffDoozan once the 2026-05-01 File Export for enwiktionary is ready, it would be great if you can confirm whether your observed differences are ameliorated. We do expect differences between File Export and the legacy XML dumps, but they should be minor.

Sorry for the delayed response. It seems this is only partially resolved. There are no longer any pages in the ContentFileExport that are missing the page text, but there are still a lot of pages missing altogether.

wget https://dumps.wikimedia.org/enwiktionary/20260601/enwiktionary-20260601-pages-meta-current.xml.bz2
wget https://dumps.wikimedia.org/other/mediawiki_content_current/enwiktionary/2026-06-01/xml/bzip2/enwiktionary-2026-06-01-p6p11719661.xml.bz2
bzcat enwiktionary-2026-06-01-p6p11719661.xml.bz2 | grep '^    <title>' | grep -v ':' | pv -l > enwiktionary-2026-06-01-p6p11719661.xml.bz2.allpages
bzcat enwiktionary-20260601-pages-articles.xml.bz2 | grep '^    <title>' | grep -v ':' | pv -l > enwiktionary-20260601-pages-articles.xml.bz2.allpages
wc -l *allpages
  8875844 enwiktionary-2026-06-01-p6p11719661.xml.bz2.allpages
  8890057 enwiktionary-20260601-pages-articles.xml.bz2.allpages

First 10 missing pages:

diff enwiktionary-2026-06-01-p6p11719661.xml.bz2.allpages enwiktionary-20260601-pages-articles.xml.bz2.allpages | grep "^>" | head

>     <title>brown</title>
>     <title>December</title>
>     <title>July</title>
>     <title>September</title>
>     <title>October</title>
>     <title>November</title>
>     <title>quid pro quo</title>
>     <title>avatar</title>
>     <title>spring</title>
>     <title>de</title>

I don't know if it's just a coincidence, but all 10 of these pages had edits on June 1st.