While most processing time might come from kafka-flink configuration, it might be worth a look to find places in mwedittypes that might be bottlenecks, especially for really large edits. Right now we are using SimpleEditTypes, hence this is less of a concern. But if StrcuturedEditType is used in the future, such profiling will most certainly be required. Nevertheless, since SimpleEditTypes will be running in production, it is best to identify issues beforehand.
Note: We are still debugging kafka-flink configuration that are the main reasons for slow streaming pipelines. See T421216 and T421026. If we find, even after the tunings are done, the edit-types pipeline is slower than we expect, this task becomes more relevant. Specifically for SimpleEditTypes.
Done:
- Profile SimpleEditType: execution time and memory. Identify bottlenecks, inefficiencies, and memory leaks
- Document findings
- Identify priority and create tasks for fixes if issues found