Consult Dumps users and other community members about the future of Dumps
Open, HighPublic
Actions

Assigned To

Authored By

	Milimetric
	May 31 2023, 7:10 PM

Description

What is the problem/what do you want to achieve?

We have multiple questions about our dumps as we work to improve them.

How can we help you?

We would like to work with the community to get answers to some questions and rough consensus on some directions we decide on. In this order:

If we move from XML to JSON Lines formatting, would you love this, hate it, need time to migrate (how much?), etc.
We can output more incremental dumps. How would you like to see this? Time chunks? What are your use cases for more incremental updates to the dumps?
Which of the many dumps artifacts that are generated do you need, and which would be very helpful but with some changes?

What does success look like?

We have decisions documented, with dissenting opinions noted, and consensus highlighted on the questions above (I think these three are good but we may add one or two more as we go).

What is your deadline?

The sooner the better for #1, but it's better to take our time and be thorough with #2 and #3.

Related Objects
Search...

Status	Assigned	Task
Invalid	VirginiaPoundstone	T345988 [Epic] XML MediaWiki data dumps for right to fork
Open	• Quiddity	T337887 Consult Dumps users and other community members about the future of Dumps
Resolved	VirginiaPoundstone	T344836 Create communications plan for data dumps transition

Event Timeline

Milimetric created this task.May 31 2023, 7:10 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2023, 7:10 PM

Milimetric added a parent task: T330296: Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark.May 31 2023, 7:11 PM

@Milimetric Thanks for filing this. While we're more than happy to help, we as a team have no idea who those users are nor can figure that out in any way different than, well, asking on generic fora, "are you a dump user". Will you give us data about them?

PS: Also please give me an idea of a timeline. Are we talking "this month", "next FY", ...? TY.

xcollazo subscribed.Jun 2 2023, 2:14 PM

• lbowmaker subscribed.Jun 5 2023, 6:11 PM

@Elitre: good point. I think we're no more knowledgeable about dumps users either. Ariel had a presentation that explained why it's hard to even know who they are (because dumps are mirrored all over the world and we can't track that usage at all). I think if we go forward with the approach we took in the Wikistats 2 consultation, where Erik Zachte asked on the mailing lists and his personal blog and the existing Wikistats 1 website about it, we might be ok. I think ultimately we involved the right people there. For reference: https://www.mediawiki.org/wiki/Wikistats_2.0_Design_Project/RequestforFeedback

As for timeline, for #1 we'd like to start right away, as soon as we figure out the right approach as per above ^. For #2 and #3, we can follow up after we start and see who engages and all that. If that answer's too wishy washy, please do ping me and we can chat in more detail.

There is a listserv that seems to be specifically for dumps users at https://lists.wikimedia.org/hyperkitty/list/xmldatadumps-l@lists.wikimedia.org/.

@Milimetric I'll come up with a plan and LYK when I can, thanks.

• Elitre claimed this task.Jun 6 2023, 3:30 PM

• Elitre triaged this task as High priority.

• Elitre moved this task from Backlog & Radar to Apr-Jun-2023 on the MoveComms-Support board.Jun 6 2023, 5:55 PM

• Elitre edited projects, added MoveComms-Support (Apr-Jun-2023); removed MoveComms-Support.

We had our first meeting today and I think I have what I need to start working on this request . Given that my team's offsite is a bit in the way, this is not going to reach the public this month.

The team LMK that there's no rush. I started a doc for them to work on the language of the questions.

• Elitre moved this task from Backlog to Started on the MoveComms-Support (Apr-Jun-2023) board.Jul 4 2023, 4:28 PM

• Elitre edited projects, added MoveComms-Support (Jul-Sep-2023); removed MoveComms-Support (Apr-Jun-2023).Jul 4 2023, 4:30 PM

• WDoranWMF added a project: Experimentation Lab.Aug 3 2023, 3:02 PM

VirginiaPoundstone moved this task from Incoming to To be discussed/To be estimated on the Experimentation Lab board.Aug 17 2023, 3:48 PM

VirginiaPoundstone added a project: Dumps 2.0.Aug 30 2023, 9:09 PM

@Elitre is there any update about this task? to transfer the info to Asana Mov Calendar

Not really, we're progressing slowly. I added what seems to be a related task now.

VirginiaPoundstone mentioned this in T345988: [Epic] XML MediaWiki data dumps for right to fork.Sep 10 2023, 1:47 AM

VirginiaPoundstone moved this task from To be discussed/To be estimated to Wikistats Backlog on the Experimentation Lab board.Sep 19 2023, 2:45 PM

Snaevar subscribed.Sep 21 2023, 9:05 PM

AutoWikiBrowser has an database scanner using the dumps. Just pointing it out, seems relevant.

JJMC89 removed a project: AutoWikiBrowser.Sep 21 2023, 9:32 PM

• Elitre removed • Elitre as the assignee of this task.Sep 25 2023, 8:30 AM

Snaevar unsubscribed.Sep 25 2023, 10:12 PM

TBurmeister subscribed.Oct 3 2023, 7:44 PM

VirginiaPoundstone moved this task from Wikistats Backlog to Radar (other teams) on the Experimentation Lab board.Oct 4 2023, 7:13 PM

• RZamora-WMF edited projects, added MoveComms-Support (Oct-Dec-2023); removed MoveComms-Support (Jul-Sep-2023).Oct 6 2023, 7:45 PM

• RZamora-WMF moved this task from Backlog to Ongoing on the MoveComms-Support (Oct-Dec-2023) board.

• RZamora-WMF assigned this task to • Quiddity.Oct 11 2023, 11:40 PM

• WDoranWMF closed subtask T344836: Create communications plan for data dumps transition as Resolved.Nov 20 2023, 6:34 PM

VirginiaPoundstone edited parent tasks, added: T345988: [Epic] XML MediaWiki data dumps for right to fork; removed: T330296: Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark.Mar 1 2024, 5:22 PM

Bamyers99 subscribed.Apr 4 2024, 1:11 PM

VirginiaPoundstone moved this task from Incoming to Epics on the Dumps 2.0 board.May 14 2024, 1:34 PM

• WDoranWMF removed projects: MoveComms-Support (Oct-Dec-2023), Experimentation Lab.Jun 10 2024, 6:17 PM

Are you planning on incorporating the Wikidata QID into this dump?
It would be great to discuss this, you can contact us at wikidata-integrations@wikimedia.de

This is following up from this ticket: https://phabricator.wikimedia.org/T197090

In T337887#10367727, @SuzanneWood-WMDE wrote:

Are you planning on incorporating the Wikidata QID into this dump?
It would be great to discuss this, you can contact us at wikidata-integrations@wikimedia.de

This is following up from this ticket: https://phabricator.wikimedia.org/T197090

Hi @xcollazo
It would be great to collaborate on getting this field into the dump - if possible can you point us in the direction of who to talk to about that? Thanks so much
wikidata-integrations@wikimedia.de

Actually I already found out who to contact, thanks!

In T337887#10386061, @SuzanneWood-WMDE wrote:

Actually I already found out who to contact, thanks!

Sorry for the delay and all the hoops to contact us. Just in case copying here:

In T197090#10382057, @Ahoelzl wrote:

@SuzanneWood-WMDE thanks for the update, please reach out to the DPE / myself regarding anything Dumps 2 (and 1) related going forward.

Consult Dumps users and other community members about the future of DumpsOpen, HighPublicActions