FWIW, I've been trying to add a simple line chart of editor interface popularity over time, but it's been unironically timing out.

Devs: it would be nice to get a second opinion on these SQL queries, especially some sanity checking of the way users and sessions are grouped.

Screencaps:

Hypothesis 4 can be followed up in T363453: Question HTML dump page order.

Sneak preview:

Sneak preview for those playing at home:

When the metrics land, they should appear on https://prometheus-eqiad.wikimedia.org/analytics/targets?search=wmde_tewu .

This will simplify how we share monitoring duty during the long-running scrape job.

Stalled waiting for WMF legal review.

Well, it could be simple after all. Articles at the end are on average twice as long (by HTML length).

In this example, the segment on the left is processing the tail articles starting at the 2.6M'th row, and on the right we're processing the first articles in the dump.

Very surprisingly to me, Hypothesis 4 seems to be the only validated theory. I haven't yet identified what makes the last articles harder to process, but the performance characteristics are almost perfectly repeatable when going back and forth between sets of articles at the beginning vs. the end of the dump. Initial articles can be processed at ~1.5k articles/s, and final articles at ~250 articles/s.

WIP on the low-level-concurrency branch will let us experiment with per-page timeouts and debugging.

This may be related to T362894: Data quality: HTML dumps contain unexplainably outdated revisions of some pages. The duplicates seem to have various revision ids, here's a set showing that the article is included three times with the same title and page id, but at different versions:

tar xzf dewiki-NS0-20240201-ENTERPRISE-HTML.json.tar.gz -O | jq 'select(.name == "10.000 B.C.") | .identifier,.version.identifier'

@BTullis Thanks for highlighting this possibility! I tried the Conda environment as you suggested and it works perfectly for our needs. Even at high concurrency, the performance seems to be the same as in the bare metal environment I had cobbled together previously.

Still seeing extreme swings in performance, following the same shape as before. Now with additional metrics:

Implemented in https://gitlab.com/wmde/technical-wishes/scrape-wiki-html-dump/-/merge_requests/122

Pulling this in because it would be nice to have, to debug the slowdown we see after the first 20 minutes or so.

Some of these packages already appeary in debmonitor:

• https://debmonitor.wikimedia.org/source-packages/erlang
• https://debmonitor.wikimedia.org/source-packages/cmake

Hmm, spot-checking is only turning up articles which were created or moved after the snapshot date.

Duplicates: each copy of a page comes with a different revid, and checking the final counts we can see that our deduplication did catch the extra copies during the aggregation step:

Bringing this task into our sprint because it has data quality implications and probably blocks scraping for the moment.

Oookay there are all kinds of things happening. Diffing the two lists, we can see that the scraper is still producing duplicates:

+1._Buch_Samuel
+1._Buch_Samuel
+1._Buch_Samuel
+1._Buch_Samuel
+1._Buch_Samuel

Deleted articles shouldn't show up in either list, so Hypothesis #2 is also looking unlikely.

ApiQueryAllPages uses the page title to carry continuation state, which is very reasonable! This would only be fooled by page renames happening during the dump interval, which is possible but not likely to add up to 1%. Hypothesis #1 is looking unlikely.

There's still a small (<1%) gap in page count. Splitting a small investigation out as T362659.