While we have practices in place for unit (and to a degree integration) tests for flink applications in CI a local envs, we lack the capability to automate integration tests on staging.

A typical integration test pipeline requires deploying an image on staging, producing messages in kafka-test and observing the output in kafka-test / staging.

As an example, this is what I did recently to test a change to mw-page-content-change-enrich:

• I built an image by pushing a dev tag.
• I deployed the dev app image on staging
• I used kafkacat to collect 331815 messages from kafka-jumbo’s page_change.v1
• I used kafkacat to produce the collected messages into kafka-test’s page_change.v1

To verify that things worked:

• I kept an an eye on logstash and grafana
• I checked that messages have been enriched in kafka-tests’s page_content_change.v1 topic
• I checked for odd errors in kafka-test’s eqiad.mw_page_content_change_enrich.error

Goals

• Data generation process should be automated.
• Outcome checks should be automated.
• We should define a frozen set of data for testing regression.