Scraper: produce spreadsheet of scraped statistics for comparing wikis
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	awight
	Apr 26 2023, 2:30 PM

Description

Read all summary.yaml files we produce, and generate a spreadsheet with columns for each field we care about (all the scalars) and a row for each wiki.

Code to review:

https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/56

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		None	T335411 Scraper: produce spreadsheet of scraped statistics for comparing wikis
Resolved		awight	T332032 Create baseline statistics for reference usage (2023)
Resolved		awight	T332057 Find efficient ORES articlequality data source
Resolved		awight	T332058 Document whether our scraper patterns are based on fictitious or real semantics
Resolved		thiemowmde	T333672 Include "group" in unique ref key
Resolved		None	T332162 Run scraper on samples from several wikis
Resolved		None	T332045 Stream input file from a tarball
Resolved		None	T332047 Script to fetch the latest HTML dump for a given wiki
Resolved		awight	T332051 Alternative output formats
Resolved		awight	T332053 Basic aggregation from intermediate format
Resolved		None	T332056 Include additional page metadata
Resolved		awight	T332159 New cloud instance and attached volume for dumps processing
Resolved		awight	T332040 Shut down our previous Cloud VPS project and create a new one
Resolved		taavi	T332773 Destroy "wmde-templates-alpha" Cloud VPS project
Resolved		Andrew	T332777 Request creation of "dump-references-processor" VPS project
Resolved		awight	T333549 Requesting Cloud VPS access to NFS mount /public/dumps
Resolved		None	T332165 Implement subdirectories for different wiki outputs
Resolved		None	T332873 Decide on file naming and organization structure
Resolved		None	T334591 Collect stats on templates inside of ref tags
Resolved		None	T332308 Support checkpoint and restart
Resolved		None	T332992 Scraper: visual progress bar
Resolved		None	T335109 Scraper: iterate over all available wiki dumps
Resolved		None	T335103 Scraper: compress samples and intermediate files with gzip
Resolved		None	T335104 Scraper: add configuration for the snapshot date
Resolved		None	T335106 Scraper: make final outputs prettier
Resolved		None	T335108 Scraper: entire pipeline should be restartable
Resolved		None	T335185 Aggregate all per-page fields in some way
Resolved		None	T335186 Scraper: compute similarity between ref bodies
Resolved		awight	T335362 Scraper: profile pipeline
Resolved	BUG REPORT	Protsack.stephan	T335761 Missing Enterprise Dumps in 2023-04-20, 2023-05-01 and 2023-05-20 runs
Resolved		None	T336672 Investigate possible corruption, or bad string handling
Resolved		None	T337449 Scraper: filter out private wikis
Resolved		None	T337450 Scraper: page-summary .gz can become corrupted after crash
Resolved		awight	T338496 Attach external volume to the scraper instance, store reports there
Resolved		None	T341593 Scraper aggregation step failing on YAML input