Commons is AFAIK the only current user of MCR.

In T384945#10502939, @Pppery wrote:

Commons is AFAIK the only current user of MCR.

Right, slot_roles tables agree:

spark.sql("""
SELECT *
FROM mediawiki_slot_roles
WHERE snapshot = '2024-11'
  AND role_name != 'main'
""").show()
                                                                                
+-------+---------+--------+-----------+
|role_id|role_name|snapshot|    wiki_db|
+-------+---------+--------+-----------+
|      2|mediainfo| 2024-11|commonswiki|
+-------+---------+--------+-----------+

Change #1118066 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[analytics/refinery/source@master] Dump all content slots

https://gerrit.wikimedia.org/r/1118066

In T384945#10531264, @pfischer wrote:

While implementing I came across a content-slot-property named origin, that obviously does not have a counterpart inside the iceberg table. According to the docs for the mariaDB column of slot_origin this property references the revision that created/updated the content slot. So it might point to the revision owning it (example), or a parent of it (example).

@xcollazo, I guess we need that field in iceberg first before moving on with content slots. Otherwise you'd have to compare SHA1s of content slots when restoring in order to determine their origin.

I do understand this field is marked as required in the Version 0.11 XML XSD, but what is still unclear is what the purpose of this field is, and why we'd want to have it in the XML Dumps.

From https://www.mediawiki.org/wiki/Manual:Slots_table#slot_origin:

slot_origin
The revision.rev_id of the revision that originated the slot's content. To find revisions that changed slots, look for slot_origin = slot_revision_id.

I guess my specific question is: why would slot_origin ever be anything other than the slot_revision_id?

Perhaps @daniel can provide some context?

In T384945#10531507, @xcollazo wrote:

I guess my specific question is: why would slot_origin ever be anything other than the slot_revision_id?

Perhaps @daniel can provide some context?

Peter explain it correctly: The value of slot_origin is the revision that last updated that slot. If there is only one slot on the page, it's always the same as slot_revision_id. But if there is multiple slots, they can differ. For instance, if you have two slots, "main" and "style", and "style" was changed in revision 3 but then "main" was changed in revision 4 and 5, slot_revision_id of the "style" slot would be 3 on revision five. Similarly slot_revision_id for the "main" slot would be different from 3 on revision three, if revision 3 only updated the "style" slot.

Got it, thanks for the example @daniel. I incorporated it on docs at mediawiki.org.

This is definitely a gap we didn't think about, and will require a backwards compatible schema change to wmf_content.mediawiki_content_history_v1.

In T384945#10544813, @xcollazo wrote:

...
This is definitely a gap we didn't think about, and will require a backwards compatible schema change to wmf_content.mediawiki_content_history_v1.

Opened T386211 to tackle the gap upstream.

Change #1195330 had a related patch set uploaded (by Xcollazo; author: Xcollazo):

[analytics/refinery/source@master] MW Dumper: Add support for Multi-content Revisions (MCR)

https://gerrit.wikimedia.org/r/1195330

Change #1195330 merged by jenkins-bot:

[analytics/refinery/source@master] MW Dumper: Add support for Multi-content Revisions (MCR)

https://gerrit.wikimedia.org/r/1195330

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1751

Draft: main: bump refinery to pickup MCR fixes for mw content file export

xcollazo closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1751

Draft: main: bump refinery to pickup MCR fixes for mw content file export

xcollazo updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1755

main: bump refinery to pickup MCR fixes on mw content file export

mforns merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1755

main: bump refinery to pickup MCR fixes on mw content file export

Ran the following as analytics to remove the existing mediawiki_content_current dump, to be rerun with the new code.

hdfs dfs -rm -r /wmf/data/exports/mediawiki_content_current/*

Rerunning Airflow DAG mw_content_xml_export_current_mid_month for 2025-10-15 now.

Confirming that commonswiki MCR output looks syntactically correct. Further semantic checks will be done as part of T407649: Compare the exported content between File Export and DumpV1.

Change #1118066 abandoned by Xcollazo:

[analytics/refinery/source@master] Dump all content slots

Reason:

Superseded by 1195330.

https://gerrit.wikimedia.org/r/1118066