Package scraper binary for Airflow job
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	awight
	Fri, Jan 16, 2:08 PM

Description

This task is done when the scraper CI produces standalone binary packaging that can be run on generic Debian bullseye and the Analytics cluster.

Investigate Conda packaging
- Conclusion: This is wrong for Elixir, our https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils#artifact-config-files workflow-utils conda-dist is specific to Python project packaging.
- Abandoned: https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/165
Prepare scraper to run under Airflow
- https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/168
- https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/169
Wire mix release with MIX_ENV=prod
Add a release release CI build step in the scraper's .gitlab-ci.yml to produce the standalone archive.
Verify the package runs on stat1010 (assumed to be similar enough to the Airflow runners).
Link the packaged file as an Airflow artifact and wire into a SimpleSkeinOperator task.

Related Objects
Search...

Status	Assigned	Task
Open	None	T396752 [Epic] Metrics for sub-referencing
Open	None	T398277 [Epic] Scraping data dumps and scraper enhancements
Open	None	T396731 Scraper: Build a new dashboard for the updated scraper data
Open	None	T412019 [Epic] Schedule scraper and aggregations as an Airflow job
Open	None	T414804 Package scraper binary for Airflow job

Event Timeline

awight created this task.Fri, Jan 16, 2:08 PM

Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptFri, Jan 16, 2:08 PM

awight removed awight as the assignee of this task.Fri, Jan 16, 2:08 PM

awight removed projects: WMDE-TechWish-Sprint-2025-12-09-Christmas-Cookie, WMDE-TechWish-Sprint-2025-11-25-Spekulatius.

awight mentioned this in T412019: [Epic] Schedule scraper and aggregations as an Airflow job.

awight updated the task description. (Show Details)Fri, Jan 16, 2:14 PM

WMDE-Fisch claimed this task.Fri, Jan 16, 2:18 PM

WMDE-Fisch moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts board.

WMDE-Fisch removed WMDE-Fisch as the assignee of this task.Fri, Jan 16, 2:36 PM

WMDE-Fisch moved this task from Doing to Sprint Backlog on the WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts board.

awight mentioned this in T414803: Write an Airflow sensor to detect Enterprise Snapshots.Fri, Jan 16, 3:09 PM

awight updated the task description. (Show Details)Mon, Jan 19, 8:22 AM

awight moved this task from Incoming to NOW on the Cite (Sub-referencing) board.Mon, Jan 19, 9:11 AM

awight claimed this task.Mon, Jan 19, 10:21 AM

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts board.

The draft MR is creating a conda-dist package now, but there is a major shortcoming: the scraper is not included. I'm asking how to get it included on Slack, we might end up with a Dockerized task instead.

awight moved this task from Doing to Epics / Watching / Stalled on the WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts board.Mon, Jan 19, 3:15 PM

awight added a project: WMDE-TechWish-Sprint-2026-01-20-Carrots-of-many-colors.Tue, Jan 20, 9:32 AM

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-01-20-Carrots-of-many-colors board.Tue, Jan 20, 10:33 AM

awight renamed this task from Produce conda-dist package for scraper to Package scraper binary for Airflow job.Wed, Jan 21, 1:03 PM

awight updated the task description. (Show Details)

awight updated the task description. (Show Details)Wed, Jan 21, 1:16 PM

awight moved this task from Doing to Tech Review on the WMDE-TechWish-Sprint-2026-01-20-Carrots-of-many-colors board.Thu, Jan 22, 8:24 AM

awight removed awight as the assignee of this task.Thu, Jan 22, 8:27 AM

awight updated the task description. (Show Details)Tue, Jan 27, 2:55 PM

awight updated the task description. (Show Details)