Page MenuHomePhabricator

data-reload: Improved dump verification
Closed, ResolvedPublic

Description

T428925 laid the ground work for dump verification. It implemented a naive approach which merely compared directory sizes. This approach is insufficient, considering that some previous dumps may be incomplete or only contain certain parts of wikidata (for example the lexemes).
The idea is to compare dump file sizes individually, ideally using patterns to match dump files.

Considering a dump consists of the following files (example):

wikidata-20260614-lexemes-BETA.ttl.gz
wikidata-20260614-lexemes-BETA.ttl.bz2
wikidata-20260614-all-BETA.ttl.gz
wikidata-20260614-all-BETA.ttl.bz2
wikidata-20260614-md5sums.txt
wikidata-20260614-sha1sums.txt

The verification script could use a set of filename patterns. For each of these patterns it could do the following:

  1. Verify that the file exists in the new dump
  2. Find the file in other dumps, get their average size and compare to the size of the latest dump
  3. Verify file integrity using the checksums in wikidata-YYYYMMDD-XXXsums.txt

AC:

  • The script uses a set of filename patterns to check that all necessary files exist in the dump (optionally configurable)
  • The script compares individual file sizes to the average of previous dumps
  • The script verifies file checksums

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Wikidata platform: Platform tools docker image bump to 0.0.3repos/data-engineering/airflow-dags!2447truegwikidata-platform/platform-tools-0.0.3main
Fixed parsing of --dump-file-pattern parameter.repos/wikidata-platform/wikidata-platform-tools!5truegfixed_verification_param_parsingmain
Customize query in GitLab

Event Timeline

trueg changed the task status from Open to In Progress.Jun 25 2026, 2:11 PM
trueg claimed this task.
trueg triaged this task as High priority.