For Airflow dags, we are using trusted-runners provided by rel-eng.

I have done part of the refactor in this change: https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/938941/2..3
including:

• Adding the unit tests on the important part of the code
• Add some new traits, and classes and rename some classes for better comprehension

What has been done in a first step:

• Custom partitioner POC
• First implementation
• Clarifying source & result expectation

1 metric that could have been useful was the number of task retries.

OK to move to Gitlab. 👍 I'm making it work first.

I have the first draft version in Gerrit.

Covered here: https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow/Developer_guide

Maybe add here: https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow/Developer_guide#Conventions_&_good_practices

3 of our dataset are now going to use canonical.countries.is_protected:

• Cassandra AQS pageview_top_percountry_daily
• Cassandra AQS pageview_top_bycountry_monthly
• Hive geoeditors_public_monthly

Documentation added here: https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow/Developer_guide

Hello @lbowmaker , for this ticket, a patch has already been proposed.

I'm proposing with those patches:

• https://github.com/wikimedia-research/canonical-data/pull/7
• https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/428
• https://gerrit.wikimedia.org/r/c/analytics/refinery/+/929723

Should we merge those CI pipeline changes to make it the standard in workflow utils?

We have three new pipelines in this MR:

• Build image
• Run tests
• Build & publish artifacts into the registry

I've a MR with:

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/417

Today we decided not to automatize the tagging process.

https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/27/commits
https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/26/commits

There is a second problem hidden behind the missing Scala lib: the Guava version mismatch between the one provided by Hadoop and the one included in eventutilities.

Squooping test is conclusive and the patch could be merged right now.

Refinery-source does not ship Scala anymore because it was included in wikihadoop, which is not included anymore.
https://archiva.wikimedia.org/#artifact-dependencies/org.wikimedia/wikihadoop/0.3-wmf1

Thanks all for the reviews. Even if the DAG is working, deciding the single source of truth for our dataset metadata could be great. Right now, its located in:

• airflow-dags/../dataset.yml
• airflow-dags/../..._dag.py
• DataHub

Do you know if there is a DB with the new schema version? It would be cool to have a place to test the import.

Here is a standardized version of the first iteration for easy use by ppl without knowledge of DataHub: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/386

Some propositions for an immediate and more useful next step:

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/382

Update: I'm emitting metadata to Kafka from an ad-hoc Airflow data lineage task. The configuration is setting up the communication with Kafka and the schema registry, Karapace. Then the metadata is well-fetched by the mce-consumer service on the DataHub side. Now I'm looking to use the detailed version of the data lineage event, containing more information than just the link upstream<>downstream.

I've checked the result on HDFS. It performs as expected.

OK to separate the migration from this task.

Bug: There is an extra systemd check making sure SUCCESS files are generated:
https://github.com/wikimedia/operations-puppet/blob/fc98a524be9be65935b8d80b506ca33af5d442b2/modules/profile/manifests/analytics/refinery/job/data_check.pp#L27

I like idea A because the conda env encapsulates all needed libs.