Scraper: destroy Cloud VPS runner instance
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	awight
	Sep 1 2023, 7:13 AM

Description

We're idling a lot of resources by holding our runner instance. This task is a milestone meant for blocking with other work that must be completed before we can destroy the virtual machine. The task is complete once the instance is deallocated.

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		None	T345411 Scraper: destroy Cloud VPS runner instance
Resolved		None	T341751 Publish dump scraper reports
Resolved		awight	T332032 Create baseline statistics for reference usage (2023)
Resolved		awight	T332057 Find efficient ORES articlequality data source
Resolved		awight	T332058 Document whether our scraper patterns are based on fictitious or real semantics
Resolved		thiemowmde	T333672 Include "group" in unique ref key
Resolved		None	T332162 Run scraper on samples from several wikis
Resolved		None	T332045 Stream input file from a tarball
Resolved		None	T332047 Script to fetch the latest HTML dump for a given wiki
Resolved		awight	T332051 Alternative output formats
Resolved		awight	T332053 Basic aggregation from intermediate format
Resolved		None	T332056 Include additional page metadata
Resolved		awight	T332159 New cloud instance and attached volume for dumps processing
Resolved		awight	T332040 Shut down our previous Cloud VPS project and create a new one
Resolved		taavi	T332773 Destroy "wmde-templates-alpha" Cloud VPS project
Resolved		Andrew	T332777 Request creation of "dump-references-processor" VPS project
Resolved		awight	T333549 Requesting Cloud VPS access to NFS mount /public/dumps
Resolved		None	T332165 Implement subdirectories for different wiki outputs
Resolved		None	T332873 Decide on file naming and organization structure
Resolved		None	T334591 Collect stats on templates inside of ref tags
Resolved		None	T332308 Support checkpoint and restart
Resolved		None	T332992 Scraper: visual progress bar
Resolved		None	T335109 Scraper: iterate over all available wiki dumps
Resolved		None	T335103 Scraper: compress samples and intermediate files with gzip
Resolved		None	T335104 Scraper: add configuration for the snapshot date
Resolved		None	T335106 Scraper: make final outputs prettier
Resolved		None	T335108 Scraper: entire pipeline should be restartable
Resolved		None	T335185 Aggregate all per-page fields in some way
Resolved		None	T335186 Scraper: compute similarity between ref bodies
Resolved		awight	T335362 Scraper: profile pipeline
Resolved	BUG REPORT	Protsack.stephan	T335761 Missing Enterprise Dumps in 2023-04-20, 2023-05-01 and 2023-05-20 runs
Resolved		None	T336672 Investigate possible corruption, or bad string handling
Resolved		None	T337449 Scraper: filter out private wikis
Resolved		None	T337450 Scraper: page-summary .gz can become corrupted after crash
Resolved		awight	T338496 Attach external volume to the scraper instance, store reports there
Resolved		None	T341593 Scraper aggregation step failing on YAML input
Resolved		awight	T344987 Stable ordering for scraper output columns
Resolved		None	T345412 Scraper: tune concurrency after changes to forking strategy
Resolved		None	T345417 Scraper: scan outputs for inconsistencies