Maniphest T396731

Scraper: Build a new dashboard for the updated scraper data
Open, Needs TriagePublic
Actions

Description

NOTE: Stalled because we will move the data into Hive and will make the dashboard afterwards in Superset. See T410719.

When we have the results from the sub-ref related scraper runs we'll need to update the google sheet with the data.

See e.g. 🖹 the old sheet

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T396752 [Epic] Metrics for sub-referencing
		Open		None	T398277 [Epic] Scraping data dumps and scraper enhancements
		Open		None	T396731 Scraper: Build a new dashboard for the updated scraper data
		Open		None	T410719 Persist scraper outputs to Hive
		Resolved		None	T412227 Rate limit EventGate output to 500 events/second
		Open		None	T413567 Discussion: scraper outputs vs. event sanitization
		Open		None	T412019 [Epic] Schedule scraper and aggregations as an Airflow job
		Resolved		awight	T413954 Set up Hive aggregation table and populate with sample data
		Resolved		awight	T414803 Write an Airflow sensor to detect Enterprise Snapshots
		Resolved		Gehel	T415667 Airflow devenv BashOperator image is lacking libssl1.1
		Resolved		awight	T414804 Package scraper binary for Airflow job
		Invalid		None	T414809 Write an Airflow sensor for scraper page summaries being fully imported to Hive
		Resolved		Tobi_WMDE_SW	T416001 Scraper should write directly to Hive
		Open		None	T417370 Cleanup: Elixir Hive adapter should generate better type structure
		Resolved		awight	T417633 Airflow devenv (WMDE) cannot see webproxy
		Declined		None	T417997 Run Spark Connect server in Analytics cluster
		Resolved		None	T418442 Finish and deploy scraper Airflow job
		Invalid		awight	T419654 Append environment for BashSensor
		Open		None	T419655 Migrate Tech Wishes scraper to gitlab.wikimedia.org
		Open		None	T413971 Create a Superset "dataset" and basic chart exposing the scraper per-wiki results
		Duplicate		WMDE-Fisch	T414057 Deduplicate page data that got logged multiple times during one run

Event Timeline

WMDE-Fisch created this task.Jun 12 2025, 9:30 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 12 2025, 9:30 AM

lilients_WMDE mentioned this in T396752: [Epic] Metrics for sub-referencing.Jun 12 2025, 1:37 PM

lilients_WMDE added a parent task: T396752: [Epic] Metrics for sub-referencing.

WMDE-Fisch moved this task from Incoming to Backlog on the Sub-referencing (product board) board.Jun 27 2025, 8:52 AM

WMDE-Fisch added a parent task: T398277: [Epic] Scraping data dumps and scraper enhancements.Jul 1 2025, 7:27 AM

WMDE-Fisch removed a parent task: T396752: [Epic] Metrics for sub-referencing.

WMDE-Fisch edited projects, added: Cite (Sub-referencing); removed: Sub-referencing (product board).

Lina_Farid_WMDE mentioned this in T398277: [Epic] Scraping data dumps and scraper enhancements.Jul 2 2025, 12:12 PM

WMDE-Fisch moved this task from Incoming to Dev-Work on the Cite (Sub-referencing) board.Jul 4 2025, 10:41 AM

awight added a project: WMDE-TechWish-Sprint-2025-11-11-Pumpkin-Crumb-Cake.Nov 11 2025, 10:05 AM

awight created subtask T410719: Persist scraper outputs to Hive.Nov 21 2025, 11:46 AM

lilients_WMDE added a project: WMDE-TechWish-Sprint-2025-11-25-Spekulatius.Nov 25 2025, 10:35 AM

lilients_WMDE moved this task from Sprint Backlog to Tech Review on the WMDE-TechWish-Sprint-2025-11-25-Spekulatius board.Nov 25 2025, 2:12 PM

lilients_WMDE moved this task from Tech Review to Sprint Backlog on the WMDE-TechWish-Sprint-2025-11-25-Spekulatius board.Nov 26 2025, 9:45 AM

lilients_WMDE moved this task from Sprint Backlog to Epics / Watching / Stalled on the WMDE-TechWish-Sprint-2025-11-25-Spekulatius board.Dec 1 2025, 9:57 AM

lilients_WMDE updated the task description. (Show Details)

awight added a project: WMDE-TechWish-Sprint-2025-12-09-Christmas-Cookie.Dec 9 2025, 10:11 AM

MareikeHeuerWMDE moved this task from Sprint Backlog to Epics / Watching / Stalled on the WMDE-TechWish-Sprint-2025-12-09-Christmas-Cookie board.Dec 17 2025, 1:50 PM

WMDE-Fisch added a project: WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts.Jan 7 2026, 9:16 AM

We could build on this https://superset.wikimedia.org/superset/dashboard/subref/?native_filters_key=43mKqRVUU8A4Y3w9n7HtIqbNEmqdowuWd8hgh2ML5kNYVftMXdCLqpLjG2uWaT_6

awight added a project: Epic.Jan 7 2026, 9:58 AM

lilients_WMDE moved this task from Sprint Backlog to Epics / Watching / Stalled on the WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts board.Jan 7 2026, 9:58 AM

awight added a subtask: T413757: Scraper: Run the "legacy" scraper for dewiki.Jan 7 2026, 9:59 AM

awight edited subtasks, added: T412019: [Epic] Schedule scraper and aggregations as an Airflow job; removed: T413757: Scraper: Run the "legacy" scraper for dewiki.

lilients_WMDE mentioned this in WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts.Jan 7 2026, 10:38 AM

awight created subtask T413971: Create a Superset "dataset" and basic chart exposing the scraper per-wiki results.Jan 7 2026, 12:28 PM

WMDE-Fisch created subtask T414057: Deduplicate page data that got logged multiple times during one run.Jan 8 2026, 8:42 AM