This is a parent task to capture efforts related to development and operation of Python based stateless streaming services.
===== Background
This Epic was informed by the following SPIKEs;
- [Stream Processing Framework Evaluations](https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Stream_Processing_Framework_Evaluation)
- [Build simple stateless service using PyFlink](https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Build_simple_stateless_service_using_PyFlink).
- [Evaluate a pyflink version of Mediawiki Stream Enrichment](https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Evaluate_a_pyflink_version_of_Mediawiki_Stream_Enrichment).
=== Goals
This Epic spans the following tasks.
- [] Flink wrappers and helper libraries should be moved into a dedicated git repo with packaging and CI.
- [] Flink wrappers and helper libraries should integrate with Table API. We should allow injection of UDFs (ideally cross language).
- [] We should provide scaffolding to bootstrap Python based services.
- [] We should provide utilities for local experimentation and unit testing. For instance, I would like to be able to inject mocked Sources/Sink and operate with local json files before rolling out to YARN.
- [] We should streamline packaging of pyflink applications, and ideally integrate with the shared Flink docker images
- [] Sideoutput error reporting should be made composable and more robust.
- [] Metrics and monitoring should be standardized.
- [] Deployment should be standardized using WMFs Deployment Pipeline.