Summary

Tl;dr:

We can easily express a stream config as jsonschema, and expose via datasets-config-service.
I am opposed to a monorepo for all configurations and suggest focusing current efforts on Airflow and Airflow-produced datasets. For integration with Metrics, Platform, and Mediawiki, I lean towards a service mesh approach. The service developed by @tchin could serve as a template.

The approach I've seen so far is really good in my opinion for Airflow DAG parametrization. IMHO it should be explicitly stated that the system we are building is the Airflow Dataset Config store/service, not just a generic configuration repository.

An Airflow GobblinOperator (which declares job settings) would serve as an API boundary between Airflow-managed datasets and Event Platform services. Its parametrization should be declared in the config store. This would also allow us to remove specific configurations in ESC (currently, a stream configuration declares when gobbling is expected to consume a stream, which would not be necessary if the requirement was specified downstream in the config store).

See the Prototype section for an example of such config.

Broader considerations

For this spike I investigated the current EventStreamConfig uses cases (beyond EventGate), tool vs dataset oriented config layout and interop and implications with k8s service deployments.

Based on my finding:

1. My preference is to keep EventStreamConfig and event streams declarations outside of a mono-repo config store. We could re-use the webservice business logic for a more general purpose database of stream configs. I will explore this concept further in https://phabricator.wikimedia.org/T361853 and Stream Registry: WIP , building atop https://gitlab.wikimedia.org/repos/data-engineering/dataset-config-store-service.

2. There will always be config bits that exist outside of DPE config store (mono) repos. k8s declares configs in deployment-charts. While we could parametrize flink applications via config store, we won’t be able to fully replace k8s (system) ones (e.g. memory allocation per pod). On k8s we also prefer to avoid HTTP calls to third party config services, so the config store should be embedded in e.g. Flink apps and merged with the helmfiles. I don’t see a lot of benefits to the added complexity. The same considerations apply for puppet. Effectively, both puppet and deployment-charts are API boundaries with other teams (SRE, mw).

3. The approach I’ve seen makes sense for parametrization of airflow operators. Effectively I see a coupling between the definition of a dataset and an airflow dag. In this sense, for DE / batch configs I think a Dataset “view” of the schema makes more sense than a tooling oriented one. Having everything in one place will help with developing tools for introspection (e.g. detect unused artifatcs deps becomes a simple graph traversal on the value files). In this scenario, an airflow GobblinOperator (that declares job settings) would act as an API boundary between Airflow managed Dataset and Event Platform services.

More thoughts can be found in https://wikimedia.slack.com/archives/C05RHK7PS6Q/p1713959682367569?thread_ts=1713956256.058079&cid=C05RHK7PS6Q

Prototype

I prototyped a (mock) config for webrequest ingestion. Code is available in gitlab https://gitlab.wikimedia.org/gmodena/config-store-poc/-/commit/37297c7af469fbc28bebf4de75236645a697ac27#783823b4ff48e4b442f3ed75eb4787e6657fc883_0_1.

This example adds a fragment schemas for

• Gobblin Job config

• Gobblin Event Platform config

Functional details are embedded in the description field of the linked schemas.

For simplicity, I added the fragments to the existing refine schema (but it’s just an example).
I materialized the schema in the following value file, that movies the ingestion bits of webrequest_frontend pipeline https://gitlab.wikimedia.org/gmodena/config-store-poc/-/blob/37297c7af469fbc28bebf4de75236645a697ac27/values/airflow/webrequest_frontend/values.yaml

$schema: /airflow/refine/1.0.0
descriptiom: >
  This a mock for code that does not exist yet.
  This value file reperesent a parametrization for the analytics/webrequest
  airflow dag. It assumes a Gobblin operator exists, that can be configured
  to produce the webrequest_raw dataset.
  This dataset config contains only business logic, the deployment of system (common)
  specific files is delegated to puppet and refinery.
datasets:
  - name: webrequest_raw
    fully_qualified_table_name: wmf_raw.webrequest
    gobblin:
      job: webrequest_frontend_rc0
      data_publisher_final_dir: /wmf/data/raw/webrequest
      mr_job_max_mappers: 48
      bootstrap_with_offset: latest
      writer_partition_timestamp_columns: dt
      event_platform:
        stream: webrequest_frontend_rc0
        is_wmf_production: true
    # - name: webrequest
    # <add config for airflow dag params>

This example assumes a GobblinOperator, that webrequest_raw will parametrize.

The example piggy backs on an EventStreamConfig endpoint. This is not strictly necessary, since Gobbling is stream agnostic. If needed, we could directly integrate Gobblin with Airflow by expressing only a subset of EventStreamConfig settings, namely, a list of topics to ingest. Gobblin does not perform schema validation, so an event schema labels are only metadata.

IMHO it should be explicitly stated that the system we are building is the Airflow Dataset Config store/service, not just a generic configuration repository.

@gmodena @JAllemandou, if this is the case, do we need an external service and datastore? The config is all in git.

It would be simpler if the config was just deployed alongside of Airflow, and we provided a python library to access it? Would that satisfy the requirements then?

In T361017#9757573, @Ottomata wrote:

IMHO it should be explicitly stated that the system we are building is the Airflow Dataset Config store/service, not just a generic configuration repository.

@gmodena @JAllemandou, if this is the case, do we need an external service and datastore? The config is all in git.

I might be fuzzy on terminology for this project, but in this case git (cli or gitlab api) is the datastore.

It would be simpler if the config was just deployed alongside of Airflow, and we provided a python library to access it? Would that satisfy the requirements then?

+1 to keep things embeddable, but it would be nice if we had programmatic ways to clean airflow (values?) caches. My understanding is that the current approach would facilitate that..

Personally I do like the idea of exposing configs via an HTTPs endpoint (e.g. similar to etc, or chartmuseum). At least for read operations, writes I'm not so sure.

we had programmatic ways to clean airflow (values?) caches. My understanding is that the current approach would facilitate that.

@JAllemandou can you explain this one? I don't know what this means or what the problem is. Thank you!

In T361017#9761303, @Ottomata wrote:

we had programmatic ways to clean airflow (values?) caches. My understanding is that the current approach would facilitate that.

@JAllemandou can you explain this one? I don't know what this means or what the problem is. Thank you!

I think Gabriele refers to the issue we're having with the way we handle variables currently in Airflow with DagProperties:
When using the DagProperties object in a dag, airflow automatically generates a variable for us, setting its values to the default values defined in the DAG. This is very useful for testing purposes as you can easily override some variable values to write to a test table for instance. The problem we've experienced more than once with the mechanism is when updating the DagProperties object in code and then deploying to production: since there are different values in the "variable" view of the DagProperties than the ones present in the code, Airflow uses the variable values, not the code ones. To overcome this, you need to either delete the existing variable, possibly loosing some important changes made previously, or manually merge in the variable the new and old values (I've never done it this way).

Having Airflow job configuration defined in a place where it can be changed without having to redeploy Airflow code is a strong gain to mitigate the issue described above, at least from the production code change perspective (it doesn't help as is with the testing aspect, we should think about this).

So, if we have CD for the config repo collocated with airflow , I don't see why we couldn't have the config-store embbeded with Airflow instead of being HTTP - what matters for us is that changes to config are automatically read by the next aiflow run without having to deploy. Nonetheless, I prefer having the config-store being a separated service from airflow for separation of concerns, and I also like the versatility of HTTP served config. One example use case for HTTP served config would be the ability to not send detailed job configuration to airflow, but rather the list of datasets to process, and then the job itself calls the config-store to get the parameters for its specific dataset. Another example would be reusing the config store for more than Airflow (EventStreamConfig replacement? :).

^ related: T348963: DagProperties don't automatically update Airflow variables

Having Airflow job configuration defined in a place where it can be changed without having to redeploy Airflow code is a strong gain to mitigate the issue described above, at least from the production code change perspective

Does using datasets config to declare the dag property overrides (or original property values?) overcome the need for a code deployment? You'd still have to deploy datasets config values, no?