Writing a service relying on the flink-app chart requires setting many different values that sometimes have to copied in different places. It might make sense for the flink-app chart to be a more opinionated regarding the WMF usages of flink. This could be done by providing more useful defaults and/or standardized values placeholders to allow interpolating values managed by puppet more easily.
The config bits that could see improvements could be:
- object storage configuration
- h/a setup with zookeeper
- kafka brokers
As of today writing a typical flink-app service requires the following setup:
app: image: docker-registry.discovery.wmnet/repos/my-team/my-job-image version: flink-1.16.2 env: # disable JEMALLOC for testing - name: "DISABLE_JEMALLOC" value: "true" job: entryClass: org.wikimedia.flink.MyJob initialSavepointPath: s3://my-app-storage-bucket/savepoints/init_20230904 jarURI: local:///opt/flink/my-job.jar upgradeMode: savepoint args: [ "/srv/app/conf/app-config.properties" ] state: running jobManager: replicas: 1 resource: memory: 1600m taskManager: replicas: 3 resource: memory: 3Gb cpu: 4 flinkConfiguration: taskmanager.numberOfTaskSlots: "4" high-availability: zookeeper high-availability.storageDir: s3://my-app-storage-bucket/zookeeper_ha_storage high-availability.zookeeper.quorum: flink-zk1001.eqiad.wmnet:2181,flink-zk1002.eqiad.wmnet:2181,flink-zk1003.eqiad.wmnet:2181 high-availability.zookeeper.path.root: "/flink" state.savepoints.dir: s3://my-app-storage-bucket/savepoints state.checkpoints.dir: s3://my-app-storage-bucket/checkpoints s3.access-key: myapp:account s3.secret-key: secret s3.endpoint: thanos-swift.discovery.wmnet s3.path.style.access: "true" config_files: app-config.properties: config_flag1: foo config_flag2: bar brokers: kafka-jumbo1009.eqiad.wmnet:9092 kafka: allowed_clusters: - jumbo-eqiad discovery: listeners: - mw-api-int-async-ro - schema - thanos-swift mesh: enabled: true
But could be simplified to something like:
app: image: docker-registry.discovery.wmnet/repos/my-team/my-job-image version: flink-1.16.2 env: # disable JEMALLOC for testing - name: "DISABLE_JEMALLOC" value: "true" job: entryClass: org.wikimedia.flink.MyJob jarURI: local:///opt/flink/my-job.jar initialSavepointPath: "/savepoints/init_20230904" upgradeMode: savepoint args: [ "/srv/app/conf/app-config.properties" ] state: running jobManager: replicas: 1 resource: memory: 1600m taskManager: replicas: 3 resource: memory: 3Gb cpu: 4 flinkConfiguration: taskmanager.numberOfTaskSlots: "4" config_files: app-config.properties: config_flag1: foo config_flag2: bar brokers: "/srv/app/conf/jumbo-eqiad.properties" # "standardized" properties file holding bootstrap hosts flink: object_store: swift_access_key: myapp:account swift_secret_key: secret swift_cluster: thanos-swift swift_bucket: my-app-storage-bucket savepoints_dir: "/savepoints" checkpoints_dir: "/checkpoints" high_availability: zookeeper_cluster: flink-eqiad zookeeper_path_root: "/flink" object_store_ha_path: "/zookeeper_path_root" kafka: allowed_clusters: - jumbo-eqiad discovery: listeners: - mw-api-int-async-ro - schema mesh: enabled: true
The above is just an example to illustrate the idea mainly to reduce repeated values and might be flawed.