T428925 laid the ground work for dump verification. It implemented a naive approach which merely compared directory sizes. This approach is insufficient, considering that some previous dumps may be incomplete or only contain certain parts of wikidata (for example the lexemes).
The idea is to compare dump file sizes individually, ideally using patterns to match dump files.
Considering a dump consists of the following files (example):
wikidata-20260614-lexemes-BETA.ttl.gz wikidata-20260614-lexemes-BETA.ttl.bz2 wikidata-20260614-all-BETA.ttl.gz wikidata-20260614-all-BETA.ttl.bz2 wikidata-20260614-md5sums.txt wikidata-20260614-sha1sums.txt
The verification script could use a set of filename patterns. For each of these patterns it could do the following:
- Verify that the file exists in the new dump
- Find the file in other dumps, get their average size and compare to the size of the latest dump
- Verify file integrity using the checksums in wikidata-YYYYMMDD-XXXsums.txt
AC:
- The script uses a set of filename patterns to check that all necessary files exist in the dump (optionally configurable)
- The script compares individual file sizes to the average of previous dumps
- The script verifies file checksums