Fix subref_reuse_count calculation
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	awight
	Jan 8 2026, 8:44 AM

Description

We currently calculate the subref_reuse_count field in way which results in obviously incorrect values such as -2, for the page
https://de.wikipedia.org/wiki/Jos%C3%A9_Nunes_(Herrscher)

This task is complete when the calculation is fixed.

Import the above page as a test fixture.
Write a test to verify that it is miscalculating the subref_reuse_count as -2.
- Yes but the count is currently 1, which is also wrong. This is probably due to recent page edits.
Fix the calculation, ~~probably by iterating through the subref markers and counting unique identifiers~~. Should be 3 for the sample page.

Implementation

Don't distinguish by valid / invalid sub-refs

Open question

How to deal with reuses counting and sub-refs?

Review

https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/167

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Tobi_WMDE_SW	T413891 Run new scraper and verify its mechanisms
		Open		None	T414058 Fix subref_reuse_count calculation

Event Timeline

awight created this task.Jan 8 2026, 8:44 AM

awight updated the task description. (Show Details)Jan 8 2026, 8:58 AM

awight updated the task description. (Show Details)

awight mentioned this in T413891: Run new scraper and verify its mechanisms.Fri, Jan 16, 12:15 PM

awight claimed this task.Mon, Jan 19, 3:20 PM

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-01-06-New-Year-Donuts board.

awight updated the task description. (Show Details)Mon, Jan 19, 3:46 PM

awight updated the task description. (Show Details)

Tobi_WMDE_SW added a project: WMDE-TechWish-Sprint-2026-01-20-Carrots-of-many-colors.Tue, Jan 20, 9:07 AM

Tobi_WMDE_SW moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-01-20-Carrots-of-many-colors board.

awight removed awight as the assignee of this task.Tue, Jan 20, 10:33 AM

awight moved this task from Doing to Sprint Backlog on the WMDE-TechWish-Sprint-2026-01-20-Carrots-of-many-colors board.

WMDE-Fisch added a project: Cite (Sub-referencing).Wed, Jan 21, 1:48 PM

WMDE-Fisch claimed this task.Wed, Jan 21, 2:15 PM

WMDE-Fisch moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-01-20-Carrots-of-many-colors board.

After looking into the issue and the code it seems, that the detection of "valid subrefs" is the reason for the confusion in the numbers. We're trying to detect "invalid" / "broken" sub-refs by looking for an error code in the ref tag. Then these refs will used to recalculate some of the numbers.

But some (sub) refs have an error code, but are still valid and good enough to be rendered.

See for example https://de.wikipedia.org/api/rest_v1/page/html/Jos%C3%A9_Nunes_(Herrscher)/262052622 there are errors in the Parsoid metadata due to some additional quotes in the Wikitext but the sub-refs are rendered fine.

So we either