ApiQueryAllPages uses the page title to carry continuation state, which is very reasonable! This would only be fooled by page renames happening during the dump interval, which is possible but not likely to add up to 1%. Hypothesis #1 is looking unlikely.

Deleted articles shouldn't show up in either list, so Hypothesis #2 is also looking unlikely.

A good next step would be to list all article titles and spot-check which are missing from the dump.

Oookay there are all kinds of things happening. Diffing the two lists, we can see that the scraper is still producing duplicates:

+1._Buch_Samuel
+1._Buch_Samuel
+1._Buch_Samuel
+1._Buch_Samuel
+1._Buch_Samuel

and there are lots of pages missing from the scraped dump:

 1943_B3
-1943_–_Kampf_um_das_Vaterland
-1943_–_Operation_Russland
 1943:_The_Battle_of_Midway
...
-Ådalens_IF
...
 A_Great_Day_in_Harlem
-A_Great_Day_in_Harlem_(Film)
-A_Great_Day_in_Harlem_(Foto)
+A_Great_Place_to_Call_Home
+A_Great_Place_to_Call_Home
+A_Great_Place_to_Call_Home
 A_Great_Place_to_Call_Home

Bringing this task into our sprint because it has data quality implications and probably blocks scraping for the moment.

Duplicates: each copy of a page comes with a different revid, and checking the final counts we can see that our deduplication did catch the extra copies during the aggregation step:

uniq dewiki-processed-normalized.txt  | wc -l
2884601

Interestingly, there are additional pages which were rejected because the final aggregate count is 2880242.

Checking whether these can be explained by page renames where we detected a duplication by pageid:

gunzip -c /srv/published/datasets/one-off/html-dump-scraper-refs/20240201/dewiki-20240201-page-summary.ndjson.gz | jq -r '.pageid' | sort | uniq | wc -l
2880242

Yes, this matches exactly so from our side the duplicates issue can be considered closed. There are still questions about why the enterprise dump includes these lines, of course...