In https://phabricator.wikimedia.org/T308017 we are introducing a new schema for page changes. Once available, this should become the source of Mediawiki Stream Enrichment.

This task wants to capture an ongoing discuss on what the output schema of the kafka mediawiki.page_content_change topic should look like. Currently it's based on a modified revision create, moving forward it should be base on https://phabricator.wikimedia.org/T308017.

Design choices to consider:

• Do we need a new schema at all, or can we add an optional content field to https://phabricator.wikimedia.org/T308017, which will be filled when needed?
• Do we want to keep all fields, or just store content and minimal metadata? Right now we are leaning towards keeping all metadata + content.
• How will data be ingested in HDFS? Right now we are leaning towards ingesting both the consolidate page change topic (in parquet) as well as the enriched version (in avro). The different file formats should satisfy different access patterns.

Side note: we might want to remane mediawiki.page_content_change (the output topic of Mediawiki Stream Enrichment`) to something more meaningful.

Success criteria

The following need grooming and might be split into dedicated tasks

• New output schema is documented and available in gerrit
• Mediawiki Stream Enrichment produces messages with the new schema