This was resolved some time ago when SRE released updated Airflow Debian package.

Maybe just change the name of the file to data_dependencies.yaml and the module to DataDependency?
And then the user would use it like data_dependency("data-dependency-name").get_sensor_for(dag)?
With this, the semantic weirdness would be solved, no?

I talked to @BTullis about obtaining a functional test environment that would mimic the real world this service would be operating in, and he kindly provided a list of things to do in order to build such an environment. The list is in the subtask ticket https://phabricator.wikimedia.org/T371994

Also, could we add a settings.xml file with the following contents to the Docker image? It's necessary for Maven release plugin to interact with GitLab:

I've considered the option of pulling from the git origin into the destination HDFS, albeit not using a systemd timer. I've actually done something similar before in previous jobs/roles, by mounting HDFS onto a local file system, but I don't think this is a viable solution for a number of reasons:

Seconded about the .test in the group ID - do we really need that? I believe the group ID should simply be org.wikimedia and then we can have the test part in the artifact ID.

In T368033#9912923, @BTullis wrote:

In some ways, the fundamental question is: do we want to move to a continuous deployment model, or do we want to retain manual deployment.

In T368033#9909148, @BTullis wrote:

My feeling is that, at our scale and given its proximity to the dse-k8s cluster, Ceph will be just fine for these DAG volumes.

Merged and applied - done

In T360968#9865611, @Ottomata wrote:

Q: Have we discussed these ideas with Release Engineering folks? They are currently working on a similar CD project, but it might be MediaWiki focused only.

This ticket has been resolved, the tasks from the ticket definition have been performed on an-launcher1002 (an-launcher1001 has been decommissioned).

@lbowmaker this task can be closed.

@lbowmaker this task can be closed.

@lbowmaker this task can be closed.

@lbowmaker this task can be closed.

We are attempting to resolve this issue in this ticket: T365382

No update still - let's wait for a bit and see what happens, the sync refreshment period might be daily as opposed to hourly.

This task can be closed as the issue has been fixed and changes to the DAG have been merged.

In T364487#9786155, @BTullis wrote:

What's the longer-term location for the log4j properties file name? Presumably we don't want to leave the file name as aqu-log4j.properties within any folder?

MR to switch to using DagProperties: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/689

The issue was in the path to the configured log4j.properties file in Airflow UI, hdfs:///user/aqu/aqu-log4j.properties was not accessible by the Airflow user analytics.

In T362699#9722376, @Aklapper wrote:

Hi @amastilovic, can you please associate one or more active project tags with this task (via the Add Action... → Change Project Tags dropdown)? That will allow to see a task when looking at project workboards or searching for tasks in certain projects, and get notified about a task when watching a related project tag. Thanks!

I need access to the following (from the wiki page you provided):

@Arnoldokoth @Dzahn thank you!