[priority] Scraper: count number of sub-references (on dewiki)
Closed, ResolvedPublic
Actions

Description

This metric is split out to speed up information gathering ahead of reporting.

Top priority is to add this metric, test it locally, then run on dewiki:

total number of sub-references across a wiki

Implementation

Add a test fixture including a sub-ref: https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/136
Add the code to recognize sub-refs and count them: https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/137
Finally, T410251: Run scraper on dewiki, to count number of sub-references

Current status

We've finished a dewiki scrape but the results need to be verified. The scraper detected 1,245 pages with subrefs but at the time of updating this task there are 5,023 pages in the category so either the analysis is failing or growth has been abrupt.

Nov 28 run outputs: https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/2025-11-28/
dewiki details: https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/2025-11-28/dewiki-summary.json
csv: https://analytics.wikimedia.org/published/datasets/one-off/html-dump-scraper-refs/2025-11-28/all-wikis-summary.csv

Number of pages on dewiki with subrefs (pages_with_subrefs_count): 1 245
Number of subrefs on dewiki (subrefs_sum): 14 548

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T396752 [Epic] Metrics for sub-referencing
Open		None	T398277 [Epic] Scraping data dumps and scraper enhancements
Resolved		None	T396729 Scraper: Add new metrics for sub-ref data
Resolved		None	T409944 [priority] Scraper: count number of sub-references (on dewiki)
Resolved		None	T410251 Run scraper on dewiki, to count number of sub-references
Resolved	Spike	None	T396720 Scraper: Use Enterprise API to retrieve dumps
Resolved		awight	T409814 Transfer ownership of Elixir MediaWiki library to WMDE Tech Wishes
Open	Spike	None	T410142 Implement Enterprise auth token refresh logic
Open		None	T403298 Provide auth-less access to Enterprise APIs from WMF Analytics cluster
Resolved		None	T410520 Wire http proxy environment vars to scraper web adapter

Event Timeline

awight created this task.Nov 12 2025, 3:54 PM

awight mentioned this in T409945: Measure sub-reference additions by individual edit.Nov 12 2025, 4:07 PM

WMDE-Fisch updated the task description. (Show Details)Nov 17 2025, 9:28 AM

lilients_WMDE moved this task from Incoming to NOW on the Cite (Sub-referencing) board.Nov 17 2025, 1:13 PM

awight renamed this task from Scraper: count number of sub-references on dewiki to Scraper: count number of sub-references (on dewiki).Nov 19 2025, 2:22 PM

awight updated the task description. (Show Details)Nov 19 2025, 2:33 PM

awight renamed this task from Scraper: count number of sub-references (on dewiki) to [prirority] Scraper: count number of sub-references (on dewiki).Nov 23 2025, 5:03 PM

awight triaged this task as High priority.

awight updated the task description. (Show Details)

awight renamed this task from [prirority] Scraper: count number of sub-references (on dewiki) to [priority] Scraper: count number of sub-references (on dewiki).Nov 24 2025, 7:53 AM

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2025-11-11-Pumpkin-Crumb-Cake board.

awight moved this task from Doing to Tech Review on the WMDE-TechWish-Sprint-2025-11-11-Pumpkin-Crumb-Cake board.Nov 24 2025, 8:39 AM

WMDE-Fisch edited projects, added WMDE-TechWish-Sprint-2025-11-25-Spekulatius; removed WMDE-TechWish-Sprint-2025-11-11-Pumpkin-Crumb-Cake, Cite (Sub-referencing).Nov 25 2025, 2:11 PM

lilients_WMDE moved this task from Sprint Backlog to Tech Review on the WMDE-TechWish-Sprint-2025-11-25-Spekulatius board.Nov 25 2025, 2:12 PM

WMDE-Fisch added projects: WMDE-TechWish-Sprint-2025-11-11-Pumpkin-Crumb-Cake, Cite (Sub-referencing).Nov 25 2025, 2:16 PM

WMDE-Fisch updated the task description. (Show Details)Nov 25 2025, 2:36 PM

Nothing to review here. The ticket is resolved when the scraper run on dewiki is done see T410251: Run scraper on dewiki, to count number of sub-references

awight updated the task description. (Show Details)Nov 28 2025, 7:41 PM

Johannes_Richter_WMDE subscribed.Dec 1 2025, 9:50 AM

awight moved this task from Epics / Watching / Stalled to Sprint Backlog on the WMDE-TechWish-Sprint-2025-11-25-Spekulatius board.Dec 1 2025, 11:28 AM

Verifying the snapshot date, I currently see the dewiki snapshot has "date_modified" => "2025-11-21T00:38:19.054220838Z" which is very recent. It would make sense if this was also the snapshot that the scraper ran against (TODO: log the snapshot date at the beginning of each run), but it doesn't explain why the discovered subref article count is so low.

Next step could be to get current subref pages directly from the category (done in T409945: Measure sub-reference additions by individual edit) and read the latest revisions. As a diagnostic, maybe compare the latest revision from the snapshot dump against revisions from the history.

awight added a project: WMDE-TechWish-Sprint-2025-12-09-Christmas-Cookie.Dec 9 2025, 10:12 AM

WMDE-Fisch closed subtask T410251: Run scraper on dewiki, to count number of sub-references as Resolved.Dec 9 2025, 10:20 AM

awight moved this task from Sprint Backlog to Done on the WMDE-TechWish-Sprint-2025-12-09-Christmas-Cookie board.Dec 11 2025, 6:48 AM

WMDE-Fisch closed this task as Resolved.Jan 6 2026, 10:53 AM