This is my attempt at a big-picture task on a topic I don't know much about so apologies if I'm reinventing the wheel or missing why such a system would be difficult. It's mostly based on our experiences with T316699: [EPIC] Growth: Positive reinforcement - Iteration 2 where we ended up building a "miniature data lake" in MediaWiki, and it was not a fun experience. (Although it's entirely possible there was a better way and we just didn't know about it - we should have talked to Data Engineering beforehand but didn't think of it.)

@Ladsgroup mentioned T309738: Move MediaWiki QueryPages computation to Hadoop which I think is a subset of this problem.

Love it, I think you describe the problem and situation well. The dumps work that we're doing right now, like this, is basically working on a more frequent sync. The overall event platform tasks about this are:

T291120: MediaWiki Event Carried State Transfer - Problem Statement and T120242: Eventually Consistent MediaWiki State Change Events as a subset. We spawned the dumps work as one instance of those thoughts there. And I'm doing the QueryPages work you linked to as a thing that can kind of be done for some queries without the more frequent data. Hopefully getting something out pushes us more towards the big picture. More on the dumps work: T330296: Dumps 2.0 Phase I: Proof of concept for MediaWiki XML content dump via Event Platform, Iceberg and Spark

Related: T258511: Data Lake incremental Data Updates

BTW I gave a state of the data platform talk at the WMF Data Strategy convening in November 2024.

There's lots of detail, but if you scroll to the end where it talks about risks, this exact problem is described.

Now what do we do about it?

See https://wikitech.wikimedia.org/wiki/MediaWiki_External_Data_Problem for more use cases.

Thanks for filing, added this to our essential work catalog.

Another possible use case:

Ideally we can eventually show aggregate stats for Mentors. This has always been part of the long-term plans for Mentorship: Mentorship impact. A community member just suggested some statistics that would be useful for Mentors in this discussion.

In T341649#10659194, @Ottomata wrote:

BTW I gave a state of the data platform talk at the WMF Data Strategy convening in November 2024.

There's lots of detail, but if you scroll to the end where it talks about risks, this exact problem is described.

Now what do we do about it?

See https://wikitech.wikimedia.org/wiki/MediaWiki_External_Data_Problem for more use cases.

I think the link should have been: https://wikitech.wikimedia.org/wiki/MediaWiki_Externalized_Data_Problem

Another couple of use cases based on some "fun" I have had with the new iceberg revision datasets is collecting metrics on the state of Wikimedia commons depicts, and instance types on wikidata and feeding that data back into mediawiki to display static pie charts and line charts showing growth in those areas over time.
This would for example allow wiki projects to see the impact they have having when adding depicts statements on masa.
And would allow wikidata to monitor the content split, particularly with bibliographic items.
One could argue some of this is possible via the query service, however the data lake is likely a better fit

Nice, @Addshore, could you summarize that in a brief one or two items and add them to the task description? Thank you!