- User Since
- Jun 9 2022, 6:42 PM (40 w, 6 d)
- LDAP User
- MediaWiki User
- XCollazo-WMF [ Global Accounts ]
Tue, Mar 21
@CBogen: Do we want to remove the SVGs only from section-level image suggestions, or from all suggestions?
Mon, Mar 20
Also the anacoda-wmf package isn't available in bullseye
Fri, Mar 17
Ok this one was fun. Explanation:
The culprit seems to be the definition of the bump_on_airflow_dags CI step. I deleted it temporarily and now MRs open up with a proper pipeline run triggered by the MR creation.
This has been merged into branch https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/tree/T311289-combined.
Thu, Mar 16
This was deployed as part of T332031. Closing.
Follow up items after Airflow 2.5.1 upgrade on platform_eng:
- Seems like we lost history for 2 DAGs. One dag does have all history. @Antoine_Quhen is this something recoverable?
Ok this has been done now.
Preemptively paused all DAGs just now.
Wed, Mar 15
Deployment is now blocked by the Airflow 2.5.1 upgrade (See T332031). We could just branch out for this deployment, but since the upgrade is slated for Thu Mar 16, it doesn't make much sense to pay the branching penalty for just one deployment.
Tue, Mar 14
Folks, on T328672, we are calling this column section_heading.
Scheduled for Thursday March 16 @ 16:00 UTC.
Mon, Mar 13
Hal needs to deploy to the platform-eng Airflow instance. So he needs platform-eng-deployers.
Fri, Mar 10
Confirmed that I can use spark3 from an-airflow1004.eqiad.wmnet:
Tue, Mar 7
Mon, Mar 6
Fri, Mar 3
One issue we have is that the pipeline ran for snapshot=2023-02-20 while we were working on this task.
Wed, Mar 1
My vote is for standardizing on Tnnn, as that is what I expect as a Phabricator user. It also avoids your ambiguity issue.
Passing by to note that you can use wmf.unique_editors_by_country_monthly today in Superset by creating a dataset on top of the Hive table. I just did this, and generated this example world map dashboard based on it: https://superset.wikimedia.org/superset/dashboard/432/.
Deployed to prod.
Tue, Feb 28
Mon, Feb 27
Opened T330667 for following up on proper sensors for DAG run order.
Fri, Feb 24
This is being taken care of via T328672.
MR up for review at https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/10.
Wed, Feb 22
The only task that remains to be done that I am unable to do is putting the conda env that this script runs on into archiva or the airflow-dags hdfs file, which I don't currently have permission to access.
@Htriedman this step is done automatically when we deploy to the production analytics instance. It gets pulled from the URI you specify on the artifacts file. So as soon as your DAG gets merged, any of the folks with admin privilege on that instance can deploy to prod on your behalf (me included).
Tue, Feb 21
Feb 16 2023
Addressed review suggestions to https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/228.
Feb 14 2023
(this is still waiting for review)
Feb 13 2023
Released new version via https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/234.
All right, added some guidance to the onboarding template around the username field when registering with Wikitech.
Feb 10 2023
Let's do a debug session if you have the time @Cparle.
Feb 9 2023
Feb 8 2023
Thanks for looking into this folks. I understand this is not possible right now, and I do use gerrit quite often, so I'll close this.
Feb 7 2023
- Merged https://gitlab.wikimedia.org/repos/structured-data/section-image-recs/-/merge_requests/2
- Opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/228 for review. Still needs a bit of work.
Feb 6 2023
- Created fork of https://gitlab.wikimedia.org/repos/research/section-image-recs at https://gitlab.wikimedia.org/repos/structured-data/section-image-recs
- Merged https://gitlab.wikimedia.org/repos/structured-data/section-image-recs/-/merge_requests/1
- Renamed default branch to main.
Feb 3 2023
Feb 1 2023
Jan 31 2023
Jan 30 2023
Jan 26 2023
Jan 24 2023
Jan 23 2023
I have left comments on the pipeline logic for possible future generalization of the solution so that other folks could benefit from it.
Partial automation has been implemented for section-topics as well via https://gitlab.wikimedia.org/repos/structured-data/section-topics/-/merge_requests/10.
Nice find @MunizaA!
Jan 20 2023
Partial automation has been implemented for image-suggestions via MR https://gitlab.wikimedia.org/repos/structured-data/image-suggestions/-/merge_requests/9.
Jan 19 2023
Jan 18 2023
Confirmed that the systemd timer is present on an-launcher1002:
xcollazo@an-launcher1002:~$ systemctl list-timers | grep drop-image-suggestions Mon 2023-01-23 13:00:00 UTC 4 days left n/a n/a drop-image-suggestions.timer drop-image-suggestions.service
The only remaining task here is the merging of https://gerrit.wikimedia.org/r/c/operations/puppet/+/870974/, which I hope will happen in the next day or so.
Just for fun:
It took ~4 hours to run! This makes sense considering the amount of partitions and files to move to the trash.
Confirmed that user analytics-platform-eng and the keytab are available on an-launcher1002: