https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/27/commits
https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/26/commits

There is a second problem hidden behind the missing Scala lib: the Guava version mismatch between the one provided by Hadoop and the one included in eventutilities.

Squooping test is conclusive and the patch could be merged right now.

Refinery-source does not ship Scala anymore because it was included in wikihadoop, which is not included anymore.
https://archiva.wikimedia.org/#artifact-dependencies/org.wikimedia/wikihadoop/0.3-wmf1

Thanks all for the reviews. Even if the dag is working, it could be great to decide the single source of truce for our datasets metadata? Right now, its located in:

• airflow-dags/../dataset.yml
• airflow-dags/../..._dag.py
• DataHub

Do you know if there is a DB with the new schema version? It would be cool to have a place to test the import.

Here is a standardized version of the first iteration for easy use by ppl without knowledge of DataHub: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/386

Some propositions for an immediate and more useful next step:

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/382

Update: I'm emitting metadata to Kafka from an ad-hoc Airflow data lineage task. The configuration is setting up the communication with Kafka and the schema registry, Karapace. Then the metadata is well-fetched by the mce-consumer service on the DataHub side. Now I'm looking to use the detailed version of the data lineage event, containing more information than just the link upstream<>downstream.

I've checked the result on HDFS. It performs as expected.

OK to separate the migration from this task.

Bug: There is an extra systemd check making sure SUCCESS files are generated:
https://github.com/wikimedia/operations-puppet/blob/fc98a524be9be65935b8d80b506ca33af5d442b2/modules/profile/manifests/analytics/refinery/job/data_check.pp#L27

I like idea A because the conda env encapsulates all needed libs.

I can confirm that now the data looks good in both:

• https://dumps.wikimedia.org/other/pageview_complete/monthly/2023/2023-03/
• https://dumps.wikimedia.org/other/pageview_complete/2023/2023-03/

The data has been regenerated and should be pushed automatically to the web endpoint at 5 am UTC.

https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/352

Optionally, some updates to the java code: https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/904778

No history was lost. Some dags have been renamed: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/commit/760f31789ee20f3e6e263fa4733ff51202fa52a0

Here is the Airflow Job: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/260

In the description, I've added a list of jobs that look like dependencies.

Note: we should create a new branch, main_airflow_2_5_1, in airflow-dags to deploy the code only one the instances which are migrated. Hopefully, it shouldn't take long. Then when the migration process is finished we could merge to main and deploy from main.