Change Details

For some Data Engineering workflows, we depend on knowing which datacenter is active and producing event data so that our pipelines can ingest it. Right now, [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/bb7526e94c3ccd9817d259c5b69df90f56b0f365/analytics/dags/wikidata/item_page_link/wikidata_item_page_link_weekly_dag.py#L100-102 | we modify the pipelines manually ]], and we invariably forget till the SLA alarms remind us. It would be nice to have an API to know what datacenter is active. Nothing fancy, just an HTTP GET that would tell me whether its `eqiad` or the like. While reviewing the new [[ https://docs.google.com/document/d/1EtOvWb3yd3A9DA6_dfxgnhQOF5lmeqA4gyzCBuBUymk/edit?disco=AAAAyiGYiHk | Datacenter Switchover Policy ]], I suggested such an API and @akosiaris quickly pointed me to an existing endpoint at `https://config-master.wikimedia.org/mediawiki.yaml` that spits out: ``` # the master datacenter for mediawiki primary_dc: eqiad # read-only settings read_only: codfw: false eqiad: false ``` What we want is the `primary_dc`. In this task we should: [x] Investigate the stability of this API. [] If deemed stable, then modify our codebase so that we don't have to manually do these changes. [] Make sure we can override if necessary.

**New description:** >>! In T338796#9421859, @xcollazo wrote: >>>! In T338796#9421258, @Ottomata wrote: >> @xcollazo do we need this anymore now that we've enabled canary events for all MW state event streams? You should be able to depend on both datacenter partitions being marked as ready, even if there are no real events in one of the DCs. > > Nice! > > In that case, what we want is to rewrite all instances in Airflow where we do `pre_partitions=['datacenter=eqiad'],` to read like `pre_partitions:=[["datacenter=eqiad", "datacenter=codfw"]]`. > > So will update description above and we can reuse the same ticket, for context. --- **Old description:** For some Data Engineering workflows, we depend on knowing which datacenter is active and producing event data so that our pipelines can ingest it. Right now, [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/bb7526e94c3ccd9817d259c5b69df90f56b0f365/analytics/dags/wikidata/item_page_link/wikidata_item_page_link_weekly_dag.py#L100-102 | we modify the pipelines manually ]], and we invariably forget till the SLA alarms remind us. It would be nice to have an API to know what datacenter is active. Nothing fancy, just an HTTP GET that would tell me whether its `eqiad` or the like. While reviewing the new [[ https://docs.google.com/document/d/1EtOvWb3yd3A9DA6_dfxgnhQOF5lmeqA4gyzCBuBUymk/edit?disco=AAAAyiGYiHk | Datacenter Switchover Policy ]], I suggested such an API and @akosiaris quickly pointed me to an existing endpoint at `https://config-master.wikimedia.org/mediawiki.yaml` that spits out: ``` # the master datacenter for mediawiki primary_dc: eqiad # read-only settings read_only: codfw: false eqiad: false ``` What we want is the `primary_dc`. In this task we should: [x] Investigate the stability of this API. [] If deemed stable, then modify our codebase so that we don't have to manually do these changes. [] Make sure we can override if necessary.

**New description:** >>! In T338796#9421859, @xcollazo wrote: >>>! In T338796#9421258, @Ottomata wrote: >> @xcollazo do we need this anymore now that we've enabled canary events for all MW state event streams? You should be able to depend on both datacenter partitions being marked as ready, even if there are no real events in one of the DCs. > > Nice! > > In that case, what we want is to rewrite all instances in Airflow where we do `pre_partitions=['datacenter=eqiad'],` to read like `pre_partitions:=[["datacenter=eqiad", "datacenter=codfw"]]`. > > So will update description above and we can reuse the same ticket, for context. --- **Old description:** For some Data Engineering workflows, we depend on knowing which datacenter is active and producing event data so that our pipelines can ingest it. Right now, [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/bb7526e94c3ccd9817d259c5b69df90f56b0f365/analytics/dags/wikidata/item_page_link/wikidata_item_page_link_weekly_dag.py#L100-102 | we modify the pipelines manually ]], and we invariably forget till the SLA alarms remind us. It would be nice to have an API to know what datacenter is active. Nothing fancy, just an HTTP GET that would tell me whether its `eqiad` or the like. While reviewing the new [[ https://docs.google.com/document/d/1EtOvWb3yd3A9DA6_dfxgnhQOF5lmeqA4gyzCBuBUymk/edit?disco=AAAAyiGYiHk | Datacenter Switchover Policy ]], I suggested such an API and @akosiaris quickly pointed me to an existing endpoint at `https://config-master.wikimedia.org/mediawiki.yaml` that spits out: ``` # the master datacenter for mediawiki primary_dc: eqiad # read-only settings read_only: codfw: false eqiad: false ``` What we want is the `primary_dc`. In this task we should: [x] Investigate the stability of this API. [] If deemed stable, then modify our codebase so that we don't have to manually do these changes. [] Make sure we can override if necessary.