Scraper: visual progress bar
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	awight
	Mar 24 2023, 1:41 PM

Description

Design and code a light UI for the scraper. Show parallel cursors moving through 0-100% of source files as a stack of progress bars. This view rolls up concurrent workers, eg. using the checkpoint counter to show overall progress, but partitions on wiki.

Print a count of rows processed, and the percentage of total lines.

Screencast demonstrating the owl library (sample code, asciicinema):

Efficient implementation depends on a missing dump stat "number of pages in Main namespace", see subtask T332858: Enterprise HTML dump stats should include file size and article count.

Code to review:
https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/51

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T345411 Scraper: destroy Cloud VPS runner instance
Resolved	None	T341751 Publish dump scraper reports
Resolved	None	T335411 Scraper: produce spreadsheet of scraped statistics for comparing wikis
Resolved	awight	T332032 Create baseline statistics for reference usage (2023)
Resolved	None	T332992 Scraper: visual progress bar
Resolved	None	T335109 Scraper: iterate over all available wiki dumps

Event Timeline

awight created this task.Mar 24 2023, 1:41 PM

awight updated the task description. (Show Details)Mar 25 2023, 8:29 PM

I'm pasting a woefully incomplete attempt to report on progress every N rows, to spark conversation.

diff --git a/lib/html_dump.ex b/lib/html_dump.ex
index ab6e9a9..0731873 100644
--- a/lib/html_dump.ex
+++ b/lib/html_dump.ex
@@ -1,4 +1,6 @@
 defmodule Wiki.HtmlDump do
+  @window_size 1_000
+
   def parse_file(path) do
     path
     |> File.stream!()
@@ -6,10 +8,17 @@ defmodule Wiki.HtmlDump do
     |> Enum.to_list()
   end
 
-  def parse_stream(%File.Stream{} = input) do
+  def parse_stream(input) do
+    window = Flow.Window.count(@window_size)
+
     input
     |> Flow.from_enumerable()
     |> Flow.filter(&(&1 != ""))
+    |> Flow.partition(window: window)
+    |> Flow.on_trigger(fn acc ->
+      IO.puts("hash")
+      {:nothing, acc}
+    end)
     |> Flow.map(&parse_line/1)
   end