What you want is a mediawiki_html_content_current like Mediawiki_content_current_v1

In T426347#11923079, @Ottomata wrote:

What you want is a mediawiki_html_content_current like Mediawiki_content_current_v1

Yes, something like that would be good, am presuming we just time travel in Iceberg there as needed. For MMD we can just query the existing event table in the interim of course (understood the table scan is a bit heavier this way).

I'm seeing about 19M enwiki pages in namespace 0. The MariaDB analytics replica for commonswiki globalimagelinks table thinks there are 5.8M distinct namespace 0 enwiki pages embedding a total of about 7M distinct Commons images (average of about 4.8 images per page going to 28.1M rows in scope for starting point). I imagine a portion of those pages are utilizing merely navigational images, and we know many lack a caption or alt text, but that's okay. There are about 130M bitmap files (excluding DjVu and XCF) in namespace 6 on commonswiki.

Is the plan to backfill the data in batch, or to force the mediawiki_page_html_content_change_v1 streaming pipeline to backfill the data?
I can think about a few different ways of doing it on streaming, but just wanted to point out that the event.mediawiki_page_html_content_change_v1 has all events in order, just like mediawiki.page_change. Adding new events will somehow break the real order, although I guess it won't be a problem if they are the latest version of each page.

@JMonton-WMF I'm sorry, I missed your question!

To answer the question: I had considered that it may be possible to induce it via streaming. The mechanics to do so and which fields could be replayed or optional, I was less sure.

I had also thought of batch backfill, but was presuming a preference for inducing the stream processing (maybe even with a new special sort of event that gets special handling somehow).

But I don't have a strong preference on the mechanics. I think one thing we want to try to avoid is inducing excessive extra parse back pressure. Presumably the latest HTML is often available without unreasonable overhead, but not sure if there's a backpressure valve in the stream processor here that automatically handles that if it becomes a problem - not that it necessarily would; Grafana could probably just be watched and a heads up with people involved in the infrastructure in case they saw anything unusual.

So an exact backfill won't be totally possible; there are many fields we pick up from MW at event time that we can't easily set at backfill time.

However, you probably don't really care that much about those fields: what you want is the html (and some page and revision details).

So to do this, I think I would:

• Spark job that iterates over page_ids in some complete Data Lake dataset, perhaps Mediawiki_content_current_v1 or perhaps mediawiki_history.
• Best effort to map from these data sources to the mediawiki/page/rendering_content_change fields.
• iterate over these records, and fetch HTML and other details from MediaWiki APIs, filling in as many details as possible.

OR: if we wanted to automate, we could do it more like mediawiki_content_history reconcilation does: if a page_id is not in the data lake dataset, emit a 'page-change-like' reconciliation event, and have a copy of the Flink HTML enrichment job that consumes reconiliation stream and emits reconciled page_html_content_change events. Then in the Data Lake you (or us), could join the main page_html_content_change event table with the reconciled event table and have all the pages in a downstream dataset.

We'd love to one day make that kind of thing a bit more automatic for SOME eventually consistent MediaWiki page revision data lake table and associated 'derived data' (like html) tables. But that is way down the road if we ever even get there. Until then! It's manual!

You know, Search has to do a similar thing: get all CirrusSearch output for all pages to regenerate search indexes. They probably have some expertise or ideas.

Sorry, missed your message @Ottomata .

Yes, the SUP stuff is excellent. Scouting of those codbases has been a part of my GardenQL's inbuilt cross-repo scouting passes (and my inbuilt knowledge from traversing the code and discussions enough to be dangerous); to your point about considering possible streams in a different conversation (that was in the context of RDF delta streams; I think I've cracked that BTW, just with weekly guarantees or at worst monthly guarantees) I think this is very germane.

Now, one intentional thing is the search bounded context can't be utilized in production of the Multimedia Metadata Dataset because its guarantees are a bit different.

Much of what happens in SUP is the sort of stuff that everyone wants to solve many times over, this just being the latest example. I think in terms of platform capabilities an HTML munger as a first class thing with its own DSL could be useful. I know your requirements gathering has been seeking for what sorts of things people want; relevant thumbnail extraction on wikis that inline images/files (for the top seven Structured Content Wikis for Enterprise, to start with; although English the heaviest hitter) and relevant metadata extraction from Commons HTML (to do what imagemetadata does, which is parse the HTML; I know you know, just stating it here for archaeological purposes) is somewhat narrow, though. So I don't think it fits the bill for a generalized pattern.

I could see the case for an "inlines something" thing that pre-structures from the Parsoid HTML, maybe, although that is still a bit specific and usually not purely tabular output in nature; it requires post-processing. Still if we had gold-level guarantees for such output some day, you know I'd be a fan - I suspect it's best if we do extraction on HTML as a generalized thing that requires more and broader requirements gathering and an OKR-based project next FY. Not there yet (I think?) so back down to solving narrowly for present FY piece. I actually in this case don't mind this going narrow. That's party of the beauty of the new HTML change stream IMO and it affords sufficient agility.

Now, to your earlier point, a daily of the HTML would be awesome, and perfect for the daily deltas of MMD; but querying over the prior day's entries is inexpensive enough, too, shouldn't fall over very often, I think.

@dr0ptp4kt please help prioritize and if you need help on this? Thanks.

Following up here: no rush at this point. It's something I wouldn't mind revisiting in the autumn if we have some dual purpose for it. I think in the interim if it's a thing that DE is positioned to tackle the utility would be there pretty soon (for MMD and many other use cases, including Edit Suggestions if I had to wager a guess); it's true for Wikipedias there's some HTML in the research namespace (as noted in the Description), although I think it's important that dataset can evolve on its own, whereas a centralized source of truth would be best.

Now, I think in an ideal world we'd get the backfills (that is, for a page ID that doesn't have an entry yet, whatever its latest HTML is) for commonswiki, then enwiki, then the rest of WME SC wikis (cywiki,, dewiki, eswiki, frwiki, idwiki, itwiki, nlwiki, ptwiki), followed by the rest. I'm mindful of the storage and scaling, of course. For MMD, I'll work with the assumption that as more HTML makes its way in, it'll just get picked up through the MMD pipeline.

Concerning the WME SC wikis there's an affinity for Multimedia Metadata Dataset because then those datasets can then be joined.

And of course WME SC has been a part of the basis for vector embeddings for semantic search on Android as well. The WME SC parsing code (Go) is on gitlab.wikimedia.org, so conceivably any linkage for MMD, or for any desire to do similar parsing from within the data lake, becomes more straightforward that way as well.

If we do this, we should do it for all wikis, not just commons and enwiki.

This would be useful for upcoming work on DYK Archive, where we need to parse HTML to fetch Did You Know facts. If we do an initial (or even rolling/regular/saneitizer style) backfill for the latest HTML revision of every page, then we can be pretty sure that we will can have all page HTML going forward. Basically, we want a version of mediawiki_content_current, but with HTML.
(cc @Snwachukwu )

In T426347#12045768, @Ottomata wrote:

If we do this, we should do it for all wikis, not just commons and enwiki.

Same! For pragmatic reasons I'd mainly suggest...

in an ideal world we'd get the backfills (that is, for a page ID that doesn't have an entry yet, whatever its latest HTML is) for commonswiki, then enwiki, then the rest of WME SC wikis (cywiki, dewiki, eswiki, frwiki, idwiki, itwiki, nlwiki, ptwiki), followed by the rest.

...now, I guess the other question would be if there are other wikis with the DYK template that should be put early as well (but again, the point being to get full coverage after the first prioritized set of wikis).