[Epic] Scraping data dumps and scraper enhancements
Open, Needs TriagePublic
Actions

Description

Scraping data dumps

The scraper helps us gain insights into how references/citations are used in wikis. We're also building some of our success metrics based on the aggregated data the scraper creates from the enterprise data dumps (which are not publicly available anymore since March 25). The data is now only available via the Enterprise API .

As a first step, we want to enhance the scraper to collect sub-ref specific data. We will then run it in a regular cadence that allows us to monitor and learn fast. Since the enterprise dumbs are available on the 1st and 20th of each month, this schedule is currently sufficient. The enterprise team has offered to discuss an alternative cadence, should we need another.

Links & resources

Steps

Get access to data: T396720: Scraper: Use Enterprise API to retrieve dumps
Enhance the scraper: T396729: Scraper: Add new metrics for sub-ref data
Run the scraper: T396730: Scraper: Run the scraper with the new features regularly
Analyze subref contents T397124: Try to categorize different subref usage types
Build dashboards: T396731: Scraper: Build a new dashboard for the updated scraper data

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T396752 [Epic] Metrics for sub-referencing
Open		None	T398277 [Epic] Scraping data dumps and scraper enhancements
Open		None	T396730 Scraper: Run the scraper with the new features regularly
Resolved		None	T396729 Scraper: Add new metrics for sub-ref data
Resolved		None	T409944 [priority] Scraper: count number of sub-references (on dewiki)
Resolved		None	T410251 Run scraper on dewiki, to count number of sub-references
Resolved	Spike	None	T396720 Scraper: Use Enterprise API to retrieve dumps
Resolved		awight	T409814 Transfer ownership of Elixir MediaWiki library to WMDE Tech Wishes
Open	Spike	None	T410142 Implement Enterprise auth token refresh logic
Open		None	T403298 Provide auth-less access to Enterprise APIs from WMF Analytics cluster
Resolved		None	T410520 Wire http proxy environment vars to scraper web adapter
Open		None	T396731 Scraper: Build a new dashboard for the updated scraper data
Open		None	T410719 Persist scraper outputs to Hive
Resolved		None	T412227 Rate limit EventGate output to 500 events/second
Open		None	T413567 Discussion: scraper outputs vs. event sanitization
Open		None	T412019 [Epic] Schedule scraper and aggregations as an Airflow job
Resolved		awight	T413954 Set up Hive aggregation table and populate with sample data
Resolved		awight	T414803 Write an Airflow sensor to detect Enterprise Snapshots
Resolved		Gehel	T415667 Airflow devenv BashOperator image is lacking libssl1.1
Resolved		awight	T414804 Package scraper binary for Airflow job
Invalid		None	T414809 Write an Airflow sensor for scraper page summaries being fully imported to Hive
Resolved		Tobi_WMDE_SW	T416001 Scraper should write directly to Hive
Open		None	T417370 Cleanup: Elixir Hive adapter should generate better type structure
Resolved		awight	T417633 Airflow devenv (WMDE) cannot see webproxy
Declined		None	T417997 Run Spark Connect server in Analytics cluster
Resolved		None	T418442 Finish and deploy scraper Airflow job
Invalid		awight	T419654 Append environment for BashSensor
Open		None	T419655 Migrate Tech Wishes scraper to gitlab.wikimedia.org
Open		None	T413971 Create a Superset "dataset" and basic chart exposing the scraper per-wiki results
Duplicate		WMDE-Fisch	T414057 Deduplicate page data that got logged multiple times during one run
Resolved		awight	T402613 Production monitoring for TechWishes scraper
Resolved		None	T412018 Make scraper robust against EventGate errors
Declined		None	T412082 Implement resumability for the scraper
Resolved		Tobi_WMDE_SW	T413757 Scraper: Run the "legacy" scraper for dewiki
Resolved		Tobi_WMDE_SW	T416303 Scraper: Run the scraper for all wikis with the January data
Resolved		Tobi_WMDE_SW	T418082 Scraper will output to a simple ND-JSON file
Resolved		None	T418864 Backfill per-wiki scraper summaries into Hive

Event Timeline

WMDE-Fisch created this task.Jul 1 2025, 7:26 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 1 2025, 7:26 AM

WMDE-Fisch added subtasks: T396720: Scraper: Use Enterprise API to retrieve dumps, T396730: Scraper: Run the scraper with the new features regularly, T396729: Scraper: Add new metrics for sub-ref data, T396731: Scraper: Build a new dashboard for the updated scraper data.Jul 1 2025, 7:27 AM

WMDE-Fisch added a subtask: T397124: Try to categorize different subref usage types.Jul 1 2025, 7:31 AM

Lina_Farid_WMDE moved this task from Incoming to Stories ready for pickup on the Sub-referencing (product board) board.Jul 1 2025, 7:44 AM

Lina_Farid_WMDE updated the task description. (Show Details)Jul 2 2025, 12:12 PM

Lina_Farid_WMDE renamed this task from Story for scraper to Scraping data dumps and scraper enhancements.Jul 2 2025, 12:15 PM

Lina_Farid_WMDE mentioned this in T396752: [Epic] Metrics for sub-referencing.

WMDE-Fisch added a project: Cite (Sub-referencing).Jul 17 2025, 10:08 AM

WMDE-Fisch moved this task from Incoming to NEXT on the Cite (Sub-referencing) board.Jul 17 2025, 10:13 AM

WMDE-Fisch moved this task from Stories ready for pickup to Dev ready for sprint on the Sub-referencing (product board) board.Aug 13 2025, 9:53 AM

awight updated the task description. (Show Details)Aug 22 2025, 5:49 AM

WMDE-Fisch moved this task from Dev ready for sprint to Waiting / Stalled stories on the Sub-referencing (product board) board.Sep 4 2025, 9:48 AM

Lina_Farid_WMDE moved this task from Waiting / Stalled stories to Stories ready for pickup on the Sub-referencing (product board) board.Oct 13 2025, 2:53 PM

awight added a project: WMDE-TechWish-Sprint-2025-11-11-Pumpkin-Crumb-Cake.Nov 11 2025, 9:51 AM

awight moved this task from NEXT to NOW on the Cite (Sub-referencing) board.

lilients_WMDE moved this task from Sprint Backlog to Watching / Stalled on the WMDE-TechWish-Sprint-2025-11-11-Pumpkin-Crumb-Cake board.Nov 11 2025, 10:04 AM

Lina_Farid_WMDE moved this task from Stories ready for pickup to Tickets in sprint on the Sub-referencing (product board) board.Nov 13 2025, 9:28 AM

WMDE-Fisch removed a subtask: T397124: Try to categorize different subref usage types.Nov 17 2025, 9:37 AM

lilients_WMDE mentioned this in WMDE-TechWish-Sprint-2025-11-25-Spekulatius.Nov 25 2025, 10:20 AM

lilients_WMDE added a project: WMDE-TechWish-Sprint-2025-11-25-Spekulatius.

lilients_WMDE moved this task from Sprint Backlog to Epics / Watching / Stalled on the WMDE-TechWish-Sprint-2025-11-25-Spekulatius board.

WMDE-Fisch renamed this task from Scraping data dumps and scraper enhancements to [Epic] Scraping data dumps and scraper enhancements.Nov 26 2025, 7:10 AM

WMDE-Fisch edited projects, added Epic; removed Story.

WMDE-Fisch added a project: WMDE-TechWish (product board).Dec 5 2025, 8:11 AM

WMDE-Fisch moved this task from Incoming to Tickets in sprint on the WMDE-TechWish (product board) board.Dec 5 2025, 8:14 AM

awight created subtask T412018: Make scraper robust against EventGate errors.Dec 8 2025, 1:38 PM

awight created subtask T412019: [Epic] Schedule scraper and aggregations as an Airflow job.Dec 8 2025, 1:51 PM

awight created subtask T412082: Implement resumability for the scraper.Dec 9 2025, 9:50 AM

awight added a project: WMDE-TechWish-Sprint-2025-12-09-Christmas-Cookie.Dec 9 2025, 10:11 AM

lilients_WMDE moved this task from Sprint Backlog to Epics / Watching / Stalled on the WMDE-TechWish-Sprint-2025-12-09-Christmas-Cookie board.Dec 9 2025, 10:12 AM

WMDE-Fisch closed subtask T396720: Scraper: Use Enterprise API to retrieve dumps as Resolved.Dec 9 2025, 10:20 AM

awight closed subtask T412082: Implement resumability for the scraper as Declined.Dec 10 2025, 3:38 PM

WMDE-Fisch created subtask T413757: Scraper: Run the "legacy" scraper for dewiki.Jan 5 2026, 11:22 AM

WMDE-Fisch closed subtask T396729: Scraper: Add new metrics for sub-ref data as Resolved.Jan 6 2026, 10:52 AM