Just opened T332820 and T332822 to take care of the remaining issues as time allows.

@CBogen: Do we want to remove the SVGs only from section-level image suggestions, or from all suggestions?

Also the anacoda-wmf package isn't available in bullseye

Ok this one was fun. Explanation:

The culprit seems to be the definition of the bump_on_airflow_dags CI step. I deleted it temporarily and now MRs open up with a proper pipeline run triggered by the MR creation.

This has been merged into branch https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/tree/T311289-combined.

In T332031#8704544, @Antoine_Quhen wrote:

No history was lost. Some dags have been renamed: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/commit/760f31789ee20f3e6e263fa4733ff51202fa52a0

So new dags were created when we deployed the last version of airflow-dags. In other words, the migration was not the problem.

Yet, there is some work to reconcile both histories if needed.

This was deployed as part of T332031. Closing.

Follow up items after Airflow 2.5.1 upgrade on platform_eng:

• Seems like we lost history for 2 DAGs. One dag does have all history. @Antoine_Quhen is this something recoverable?

Ok this has been done now.

Preemptively paused all DAGs just now.

In T310541#8698694, @odimitrijevic wrote:

@xcollazo @Antoine_Quhen Does this apply to the airflow ci/cd that DE manages? Are there any improvements that we wish to adopt or expand?

If not, is this ticket still valid?

Deployment is now blocked by the Airflow 2.5.1 upgrade (See T332031). We could just branch out for this deployment, but since the upgrade is slated for Thu Mar 16, it doesn't make much sense to pay the branching penalty for just one deployment.

In T311825#8697862, @mfossati wrote:

In T311825#8688274, @Cparle wrote:

I propose that we just add this to the image-suggestions DAG rather than it having its own DAG. @mfossati @xcollazo @matthiasmullie what do you think?

Sounds good to me.

Folks, on T328672, we are calling this column section_heading.

Scheduled for Thursday March 16 @ 16:00 UTC.

Hal needs to deploy to the platform-eng Airflow instance. So he needs platform-eng-deployers.

Confirmed that I can use spark3 from an-airflow1004.eqiad.wmnet:

In T327970#8673370, @Ottomata wrote:

Seems like we don't have a robust mechanism to share secrets. Airflow does provide hooks so that we can integrate secrets in, see https://airflow.apache.org/docs/apache-airflow/2.1.4/security/secrets/secrets-backend/index.html#configuration.

@xcollazo we do have the ability to do this with our puppet stuff!
See docs here and here.

In T327970#8673139, @EBernhardson wrote:

In T327970#8671765, @Ottomata wrote:

Re https://gerrit.wikimedia.org/r/894740, we should ask @mforns @Milimetric @JAllemandou about this. I think there might be a better way? Maybe the logic to get MW db hostnames and ports should be moved out of refinery python? Or, wmfdata-python uses the refinery bin/analytics-mysql CLI. Perhaps the whole thing should move out of refinery so it is installable without deploying refinery?

Hmm, in our case what we need is the ability to ship this information in a job into the yarn cluster. We use a tiny bit of python to push both the mysql credentials file and the .dblist files from mediawiki-config into the cluster and then have custom pyspark that processes them. Moving things out of refinery might help, but we don't use refinery here anywas.

Also Q. Would it be useful to have these set up as airflow connections? Our airflow puppet has the ability to configure airflow connections.

Not really, at least not for our use case. Airflow connections are about having airflow talk directly to those things, but here we have a job that runs inside spark and uses spark's mysql connector.

One issue we have is that the pipeline ran for snapshot=2023-02-20 while we were working on this task.

Merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/253.

My vote is for standardizing on Tnnn, as that is what I expect as a Phabricator user. It also avoids your ambiguity issue.

Passing by to note that you can use wmf.unique_editors_by_country_monthly today in Superset by creating a dataset on top of the Hive table. I just did this, and generated this example world map dashboard based on it: https://superset.wikimedia.org/superset/dashboard/432/.

In T330792#8658047, @brennen wrote:

This syntax is also supported right now:

Bug: #330688

Deployed to prod.

Opened T330667 for following up on proper sensors for DAG run order.

This is being taken care of via T328672.

MR up for review at https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/10.

The only task that remains to be done that I am unable to do is putting the conda env that this script runs on into archiva or the airflow-dags hdfs file, which I don't currently have permission to access.

@Htriedman this step is done automatically when we deploy to the production analytics instance. It gets pulled from the URI you specify on the artifacts file. So as soon as your DAG gets merged, any of the folks with admin privilege on that instance can deploy to prod on your behalf (me included).

Addressed review suggestions to https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/228.

In T323036#8621701, @mfossati wrote:

It's merged, and the bot agrees! 😄 Closing.

(this is still waiting for review)

Released new version via https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/234.

All right, added some guidance to the onboarding template around the username field when registering with Wikitech.

Let's do a debug session if you have the time @Cparle.

In T329057#8598415, @bd808 wrote:

In T329057#8596354, @xcollazo wrote:

IIRC, I didn't create my LDAP user while onboarding. Do we know who this feedback should go to?

I would love to know who created a Developer account for you if you did not create it yourself. I ask that as the account creation log indicates that the account was created directly via Wikitech as a new account and not via any system for tracking people creating accounts for others: https://wikitech.wikimedia.org/w/index.php?title=Special:Log&logid=953193.

Thanks for looking into this folks. I understand this is not possible right now, and I do use gerrit quite often, so I'll close this.

• Merged https://gitlab.wikimedia.org/repos/structured-data/section-image-recs/-/merge_requests/2
• Opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/228 for review. Still needs a bit of work.

• Created fork of https://gitlab.wikimedia.org/repos/research/section-image-recs at https://gitlab.wikimedia.org/repos/structured-data/section-image-recs
• Merged https://gitlab.wikimedia.org/repos/structured-data/section-image-recs/-/merge_requests/1
• Renamed default branch to main.

Merge of
https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/15
and
https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/17

In T314592#8572667, @odimitrijevic wrote:

@xcollazo Does @JEbe-WMF need to create a similar request?

In T323107#8552347, @MunizaA wrote:

@xcollazo I think the reason why Spark2 does not push this filter down is because it does not infer filters from generators as the optimizer rule InferFiltersFromGenerate in Spark 3.1.2 does not seem to exist in Spark 2.4.4

Great, thanks!

I have left comments on the pipeline logic for possible future generalization of the solution so that other folks could benefit from it.

Partial automation has been implemented for section-topics as well via https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/10.

Nice find @MunizaA!

Partial automation has been implemented for image-suggestions via MR https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/9.

In T324125#8530275, @mfossati wrote:

@xcollazo , what about setting paths with VariableProperties, pretty much as we do with the conda artifact?
Something like helper = var_props.get('helper', '/path/to/hdfs')

Confirmed that the systemd timer is present on an-launcher1002:

xcollazo@an-launcher1002:~$ systemctl list-timers | grep drop-image-suggestions
Mon 2023-01-23 13:00:00 UTC  4 days left         n/a                          n/a                 drop-image-suggestions.timer                              drop-image-suggestions.service

The only remaining task here is the merging of https://gerrit.wikimedia.org/r/c/operations/puppet/+/870974/, which I hope will happen in the next day or so.

Just for fun:

It took ~4 hours to run! This makes sense considering the amount of partitions and files to move to the trash.

Confirmed that user analytics-platform-eng and the keytab are available on an-launcher1002: