Best SQL Example here. Will be much better with a catalog.

I don't love how weird this is to do in SQL, with the nested query and UDF based on the content_slots map field, instead of the lower level just content_body. That experiment made me think that staying in Python is going to be easier than focusing on full SQL support.

A pyflink implementation of Mediawiki Stream Enrichment has been developed and deployed on YARN. While this implementation did not write to a kafka topic directly, all enriched messages (48 hours worth of data) passed jsonschema validation. The python implementation has feature parity with the Scala one. In particular:

• It is built atop the DataStream API and operates on DataFrame[Row] (note that Row here is a pure python object and not a JVM one).
• Errors are reported to a sideoutput of String type.
• Http client implements retry logic with backoff.
• Latency, resource consumption and GC footprint are similar between the two implementations.

Relevant code:

• Enrichment application: https://gitlab.wikimedia.org/-/snippets/42
• Flink boilerplate: https://gitlab.wikimedia.org/-/snippets/44

A writeup of the outcome of this spike can be found at https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Evaluate_a_pyflink_version_of_Mediawiki_Stream_Enrichment. Feedback is welcome on the discussion page.