Splitting off from parent task.
In T355594#10973563, @SD0001 wrote:I have been poking away at a port of the list generation code that can be run on a Cloud Services instance. I ported the PHP code into Node.js – it's at https://gitlab.wikimedia.org/sd/global-election-list-builder. I initially used SQLite to store the per-wiki edit counts, but it was rather slow, processing only 60–80 users/second. I switched to redis (valkey 8.0 to be precise), which improves throughput to 150 users/second. The EditCountStore was made an interface into which any data store can be easily plugged in.
As it's querying the wiki replicas, I also did some optimizations that probably won't be possible with prod dbs:
- instead of fetching the list of global users in batches of 1000, I use a single query that fetches all 78 million users and streams it to a file. Remarkably, this takes just 37 seconds.
- the php script fetches the actor_id for each user_id individually. Instead, I similarly prefetch all user_ids + actor_ids upfront and stream it to a file. A user_editcount > 0 condition is applied which cuts the number of users from 41 million on enwiki to just 14 million.
- parallelized the queries for counting the "long edits" and "short edits", which can't be done in PHP as it's single-threaded.
Tests were run on a g4.cores16.ram32.disk20 instance belonging to Quarry project. I estimate it would take about a day to process enwiki's users. As that's s1, and we can process other db slices in parallel, it would take one day overall. And then a couple of hours for the make-global-list script to sum up counts from all wikis and produce the final list.
@jrbs This is faster than the existing scripts I think? If there's interest, I can set up a dedicated VPS project, finish up the testing and help running this for the next election.
In T355594#10981677, @jrbs wrote:In T355594#10973563, @SD0001 wrote:...
I'm not really technical enough to evaluate this but it does sound extremely promising. I spoke with Tim some years ago about a similar project.
In the meantime I have filed T398900 for this year's election since it will begin quite soon (August 27, 2025)