Duplicate articles in snapshot dump
Open, HighPublic
Actions

Assigned To

None

Authored By

	ROdonnell-WMF
	Dec 26 2023, 12:13 PM

Description

User Story: “As a client, I want to unique articles in out snapshot dumps, so that I don't need to deal with de-duplication.”

Acceptance criteria

Write an integration test to check there there are no duplicate articles in our snapshot dump output files

ToDo

Download snapshot dump files from our s3 bucket
write code that looks for duplicate articles in our enterprise_html/runs/20231201/enwiktionary-NS0-20231201-ENTERPRISE-HTML.json.tar.gz
run duplicate test on a sample of our snapshot dumps

Checklist for testing

No duplicates in future snapshot dumps for namespace 0 or other namepsaces

Things to consider:

Zendesk ticket: https://wikimediaenterprise.zendesk.com/agent/tickets/521
Follow up with client on Zendesk or by email, to tell them defect is resolved

Related Objects

Mentioned In: T353468: Add an errata to published "v1" scraper data
T354369: Scraper: sanity check dump size by including the sitestats article count
Mentioned Here: T362894: Data quality: HTML dumps contain unexplainably outdated revisions of some pages

Event Timeline

ROdonnell-WMF triaged this task as High priority.Dec 26 2023, 12:13 PM

ROdonnell-WMF created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 26 2023, 12:13 PM

Reedy updated the task description. (Show Details)Dec 26 2023, 2:36 PM

JArguello-WMF added a subscriber: AMuller-WMF.Jan 4 2024, 4:53 PM

awight mentioned this in T354369: Scraper: sanity check dump size by including the sitestats article count.Jan 4 2024, 5:29 PM

Please test for duplicates by page_id as well as revision_id. I've found duplicate revisions in the dumps and also multiple revisions of a single page.

awight mentioned this in T353468: Add an errata to published "v1" scraper data.Jan 4 2024, 5:56 PM

Protsack.stephan moved this task from Incoming to Engineering Backlog (DevOps, Maintenance, Tech debt) on the Wikimedia Enterprise board.Jan 10 2024, 3:10 PM

jberkel subscribed.Jan 26 2024, 5:50 PM

This may be related to T362894: Data quality: HTML dumps contain unexplainably outdated revisions of some pages. The duplicates seem to have various revision ids, here's a set showing that the article is included three times with the same title and page id, but at different versions:

tar xzf dewiki-NS0-20240201-ENTERPRISE-HTML.json.tar.gz -O | jq 'select(.name == "10.000 B.C.") | .identifier,.version.identifier'

page id	revision id
3394140	234268857
3394140	241670834
3394140	241670882

Duplicate articles in snapshot dumpOpen, HighPublicActions