Migrate format of pageview dumps to replace two or more consecutive zeros by char caret and then the count of the zeros
Closed, DeclinedPublicFeature
Actions

Assigned To

None

Authored By

	Dusan_Krehel
	May 30 2025, 9:10 AM

Description

Feature summary:
Dumps:

convert(ing) from old format [1] into D0CMF.

Disadvantages of the current format:

interval with a maximum of 31 values,
now, If user wanna data about the one wiki so he must download the data about all wikis.

Benefits of new format would be:

Better user comfort,
possibility grouping data (titles by local wiki),
low data to store,
low data to traffic.

D0CMF:

Analyse of store size (https://meta.wikimedia.org/wiki/User:Du%C5%A1an_Krehe%C4%BE#d0cmf).
There is code (by Dušan Kreheľ) that converts and merges pageview statistics. Tested (only user pageviews): https://archive.org/search?query=subject%3A%22d0cmf%22

{1} https://dumps.wikimedia.org/other/pageview_complete/readme.html, section "Data format".

Event Timeline

Dusan_Krehel created this task.May 30 2025, 9:10 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 30 2025, 9:10 AM

Migration of pageview dumps to D0CMF

Migration from what currently?

plus new dump split do files (§1) by local wikipedia, merge (§2) by day, month or year and number pageview data store (§3) in D0CMF.

Sorry, I cannot parse this sentence.

Benefits:

What are the disadvantages?

ex. separe local wiki,

I do not know what that means.

Dusan_Krehel updated the task description. (Show Details)May 30 2025, 3:53 PM

Pppery added a project: Dumps-Generation.May 30 2025, 7:56 PM

@Dusan_Krehel: This implies that all external software reading these Pageview datadumps needs to be rewritten to handle the proposed new format?

Aklapper renamed this task from Migration of pageview dumps to D0CMF to Migrate format of pageview dumps to replace two or more consecutive zeros by char caret and then the count of the zeros.May 30 2025, 7:57 PM

Aklapper changed the task status from Stalled to Open.

@Aklapper Yes, it's require a software update for consumer dump software.

xcollazo added a project: Data-Engineering.Jun 3 2025, 8:30 PM

@Dusan_Krehel Thank you for sharing the D0CMF pageview dump format proposal. I appreciate the detailed analysis of the current format's limitations and the potential benefits of the new approach.

After reviewing the proposal, I don't think this is something we can move forward with at this time. While the concept of per-wiki downloads and improved storage efficiency has merit, there are several considerations that make this challenging to implement in our current roadmap:

The migration would require significant coordination across multiple teams, substantial testing with our existing user base, and careful planning around backwards compatibility. * * Given our current priorities and resource constraints, we're not in a position to take on a project of this scope.

I'd encourage you to continue developing the format and gathering feedback from the community. If there's strong user demand and the technical approach matures further, this could be something to revisit in future planning cycles.

Thank you again for the thoughtful proposal and the work you've put into analyzing this problem.

Migrate format of pageview dumps to replace two or more consecutive zeros by char caret and then the count of the zerosClosed, DeclinedPublicFeatureActions

Description

Event Timeline

Migrate format of pageview dumps to replace two or more consecutive zeros by char caret and then the count of the zeros
Closed, DeclinedPublicFeature
Actions