@Danilo: You should use the bz2 compressed version of the file, they are complete (I checked). The availability of the uncompressed version seems a bug.

Compressed files are available on the computation machine stat1007:/srv/dumps/pagecounts-ez/projectviews for 2020-08, with the same size as the one in the dumps-website, but there are no uncompressed files there. I assume the uncompressed data gets computed in a raw file in the folder, copied while it's there by rsync, and once the generation job is done the data is compressed and the raw file deleted. This can lead to incorrect data being copied depending on data-generation timing and copy timing.

follow up: dumps::web::fetches::stat_dumps might be changed to force the copy only of the files that we want, so temporary ones are not transferred to labstore nodes..

I will make my script use the bz2 file when the uncompressed file is not complete.

But maybe it is also a good idea someone decompress the complete bz2 file that is in the same folder to replace the incomplete uncompressed one, it can fix the problem for the last month while the bug that caused the problem is not fixed.

Things to discuss/fix:

• Yearly projectcounts/projectviews
  • The dumps site references this page: https://dumps.wikimedia.org/other/pagecounts-ez/projectcounts/ - which contains yearly files from 2008 to 2016.
  • The rsync source folder for similar data is named projectviews (not projectcounts), leading to this page https://dumps.wikimedia.org/other/pagecounts-ez/projectviews/, not referenced on the description page

I assume the 2 pages reference data generated using 2 different datasets and therefore could exist in different places. However such a difference is not made for hourly-pagecounts (see https://dumps.wikimedia.org/other/pagecounts-ez/merged/). Shall we merge both projectviews and projectcounts in the same folder to mimic pagecounts.

• rsync only bz2 files in the merged folder - currently some not-yet compressed files are synced (possibly partially) onto the destination.

rsync example command (using / at the end source and dest as it is explicitly set on puppet):

rsync --dry-run -v -rt --include '/projectviews/' --include '/projectviews/**' --include '/merged/' --include '*/' --include '*.bz2' --exclude '*' /srv/dumps/pagecounts-ez/ /home/joal/test_rsync_pageview/