Scraper: Run the scraper for all wikis with the January data
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	WMDE-Fisch
	Feb 3 2026, 8:06 AM

Project Tags

Referenced Files

None

Subscribers

Description

Context

We want to get the next data point for our analysis of the sub-ref usage. That's data from January then. Because we also want to have baseline data for future deployments this time we want to run it for all wikis.

Task

Run the scraper for the dump that's published at the beginning of February
Make sure aggregations can be put into the database for Superset analytics

Tech notes

Since the Airflow job isn't ready yet, we'll have to run this job manually. Ideally the code review for T416001: Scraper should write directly to Hive is complete by this point.

Related Objects
Search...

Status	Assigned	Task
Open	None	T396752 [Epic] Metrics for sub-referencing
Open	None	T398277 [Epic] Scraping data dumps and scraper enhancements
Resolved	Tobi_WMDE_SW	T416303 Scraper: Run the scraper for all wikis with the January data
Resolved	Tobi_WMDE_SW	T418082 Scraper will output to a simple ND-JSON file

Event Timeline

WMDE-Fisch created this task.Feb 3 2026, 8:06 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2026, 8:06 AM

WMDE-Fisch moved this task from Incoming to Dev ready for sprint on the WMDE-TechWish (product board) board.Feb 3 2026, 8:06 AM

WMDE-Fisch moved this task from Incoming to NEXT on the Cite (Sub-referencing) board.

Tobi_WMDE_SW added a project: WMDE-TechWish-Sprint-2026-02-03-Tomatoes-of-many-colors.Feb 3 2026, 9:52 AM

Tobi_WMDE_SW mentioned this in WMDE-TechWish-Sprint-2026-02-03-Tomatoes-of-many-colors.Feb 3 2026, 10:01 AM

WMDE-Fisch renamed this task from Scraper: Run the scraper for dewiki with the January data to Scraper: Run the scraper for all wikis with the January data.Feb 9 2026, 10:48 AM

WMDE-Fisch updated the task description. (Show Details)

WMDE-Fisch moved this task from Dev ready for sprint to Tickets in sprint on the WMDE-TechWish (product board) board.Feb 10 2026, 9:42 AM

Lina_Farid_WMDE moved this task from NEXT to NOW on the Cite (Sub-referencing) board.Feb 12 2026, 10:47 AM

We agreed to do a manual run before Feb 21st.

lilients_WMDE added a project: WMDE-TechWish-Sprint-2026-02-17-Beautiful-Beetroots.Feb 17 2026, 3:34 PM

awight updated the task description. (Show Details)Feb 18 2026, 7:40 AM

awight updated the task description. (Show Details)

awight claimed this task.Feb 18 2026, 1:02 PM

awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2026-02-17-Beautiful-Beetroots board.

Running into two issues which strangely didn't appear on the test server.

Cannot cast empty objects: map() and array() have implicit map<string,string> and array<string> types which don't match the table schema. In Spark SQL I was using cast(map() as map<string,int>) and cast(array() as array<int>) but this syntax isn't allowed by the production Hive.

Multi-row insert... values... syntax isn't allowed when map() or array() are present. These are considered functions despite holding literal contents. We may have to use single-row insert..select.. for now.

Paving over the above errors by sending complex types as strings for the moment.

I've downloaded and stored the 2026-02-02 snapshot chunks on stat1010 and created a new input mode that will allow the scraper to read from those files. This makes it possible to finish processing even after the snapshot is replaced with newer revisions.

The output stage is still incomplete, since the HiveServer2 approach was too slow to be usable. After discussing with Data Engineering I'll save outputs to a temporary JSON file, details will be documented in a new subtask.

awight created subtask T418082: Scraper will output to a simple ND-JSON file.Feb 23 2026, 8:13 AM

This is the command line I used to process the saved chunks, in my home directory on stat1010:

mix scrape --dir ~/dewiki-chunks-2026-02-02/ --output=dewiki-2026-02-02-page-summary.ndjson

The output has 3.158.101 lines which is higher than the stats API tally of 3.099.970 articles. Aggregation will deduplicate by page.

Data has landed in wmde.wiki_page_cite_references_raw (per-page) and wmde.wiki_page_cite_references_monthly (totals are in one row for dewiki).

Dumping the summary here before I sign out for the day,

dbname dewiki
snapshot_date 2026-02-02 
identical_refs_count 194973
identical_refs_on_pages_with_25_or_less_refs_average 194972.95
identical_refs_on_pages_with_over_25_refs_average 0.7808942
identical_refs_on_pages_with_over_25_refs_count 93687
list_defined_ref_per_page_having_ref 0.36901948
list_defined_ref_sum 731064
max_ref_reuse_average 2.8919237
nested_ref_sum 578
page_count 3093332
pages_with_automatically_named_refs_count 116612
pages_with_identical_refs_and_over_25_refs_count 25736
pages_with_identical_refs_count 91934
pages_with_multiple_reflists_count 29991
pages_with_named_refs_count 896769
pages_with_nested_refs_count 243
pages_with_over_25_refs_count 119974
pages_with_ref_reuse_count 7024761981077
pages_with_refs_count 248833
pages_with_similar_refs_count 5986
pages_with_subrefs_count 0.5766826
proportion_of_named_refs_uniquely_named_average 0.04640607
proportion_of_pages_with_identical_refs 1.2266055E-4
proportion_of_pages_with_nested_refs 0.12560491
proportion_of_pages_with_similar_refs 0.6404346
proportion_of_pages_with_refs 0.079681166
proportion_of_refs_from_transclusion 0.3740749
proportion_of_refs_having_transclusion 0.26614386
proportion_of_refs_named_average 0.118505105
proportion_of_refs_reused_average 0.7054597
ref_by_transclusion_average 1397570
ref_by_transclusion_count 17539527
ref_count 5.6701083
ref_count_per_page 8.853531
ref_count_per_page_having_ref 2015523
reflist_count 1.0173875
reflists_per_page_having_ref 5475486
refs_with_solely_transclusion_count 6561097
refs_with_transclusions_countsimilar_refs_count 1038935
subrefs_sum 62401
transclusion_average 10.077638
transclusion_sum 31173480
wikitext_length_average 6913.2437

Also look in the secondary tables to find eg. template statistics, see the names under create_table* in https://gitlab.wikimedia.org/repos/wmde/analytics/-/tree/809d0668f2173a5655923f538cc331c976002e11/hql/airflow_jobs/wiki_page_cite_references

Tobi_WMDE_SW moved this task from Demo to Done on the WMDE-TechWish-Sprint-2026-02-17-Beautiful-Beetroots board.Mar 3 2026, 9:22 AM

Tobi_WMDE_SW closed this task as Resolved.Mar 5 2026, 9:50 AM

Tobi_WMDE_SW claimed this task.

Tobi_WMDE_SW closed subtask T418082: Scraper will output to a simple ND-JSON file as Resolved.

Scraper: Run the scraper for all wikis with the January dataClosed, ResolvedPublicActions