Page MenuHomePhabricator

Scraper: Run the scraper with the new features regularly
Open, Needs TriagePublic

Description

Context

When we made sure that the scraper can be executed with the new infrastructure to get dumps T396720: Scraper: Use Enterprise API to retrieve dumps and finished adding new aggregations to it for subrefs T396729: Scraper: Add new metrics for sub-ref data we want to get a new dataset.

Task
  • Run the scraper on a regular basis to have baseline and see progression over time
Note

This should be independent of further scraper improvements, we should just use the chance to run it regularly so we can use the data that's there with out losing the chance.

Event Timeline

WMDE-Fisch renamed this task from Scraper: Run the scraper with the new sub-ref to Scraper: Run the scraper with the new features.Jun 17 2025, 6:34 AM
WMDE-Fisch updated the task description. (Show Details)
WMDE-Fisch renamed this task from Scraper: Run the scraper with the new features to Scraper: Run the scraper with the new features on wikis before and after sub-ref.Nov 26 2025, 10:39 AM
WMDE-Fisch updated the task description. (Show Details)
WMDE-Fisch renamed this task from Scraper: Run the scraper with the new features on wikis before and after sub-ref to Scraper: Run the scraper with the new features regularly.Nov 26 2025, 10:42 AM
WMDE-Fisch updated the task description. (Show Details)