Page MenuHomePhabricator

Create voter lists on VPS instead of production
Open, Needs TriagePublic

Description

Splitting off from parent task.

I have been poking away at a port of the list generation code that can be run on a Cloud Services instance. I ported the PHP code into Node.js – it's at https://gitlab.wikimedia.org/sd/global-election-list-builder. I initially used SQLite to store the per-wiki edit counts, but it was rather slow, processing only 60–80 users/second. I switched to redis (valkey 8.0 to be precise), which improves throughput to 150 users/second. The EditCountStore was made an interface into which any data store can be easily plugged in.

As it's querying the wiki replicas, I also did some optimizations that probably won't be possible with prod dbs:

  • instead of fetching the list of global users in batches of 1000, I use a single query that fetches all 78 million users and streams it to a file. Remarkably, this takes just 37 seconds.
  • the php script fetches the actor_id for each user_id individually. Instead, I similarly prefetch all user_ids + actor_ids upfront and stream it to a file. A user_editcount > 0 condition is applied which cuts the number of users from 41 million on enwiki to just 14 million.
  • parallelized the queries for counting the "long edits" and "short edits", which can't be done in PHP as it's single-threaded.

Tests were run on a g4.cores16.ram32.disk20 instance belonging to Quarry project. I estimate it would take about a day to process enwiki's users. As that's s1, and we can process other db slices in parallel, it would take one day overall. And then a couple of hours for the make-global-list script to sum up counts from all wikis and produce the final list.

@jrbs This is faster than the existing scripts I think? If there's interest, I can set up a dedicated VPS project, finish up the testing and help running this for the next election.

...

I'm not really technical enough to evaluate this but it does sound extremely promising. I spoke with Tim some years ago about a similar project.

In the meantime I have filed T398900 for this year's election since it will begin quite soon (August 27, 2025)

Event Timeline

For T403809, I wrapped up the script and ran the first part of it (populate-edit-count). Overall, it was very successful.

  • Setup: 8 node.js processes running in parallel - one for processing each db section (s1-s8).
  • Total time taken: 9 hours, 6 minutes. s2-s8 each finished in < 3 hours. It was just s1 (enwiki) which kept running after that.
    • Average processing speeds: 430 users/second (enwiki), 640 users/s (dewiki), 609 users/s (metawiki)
  • Use of wiki replicas:
    • 2 connections for each db section (for 2 concurrent queries).
    • 1 slow query per wiki to list ids of users with non-zero edit count. This took 150 seconds for enwiki, is near-instant for small wikis.
      • This failed on two wikis (testwiki and commonswiki) - see below.
    • 2 fast db queries per user per wiki, so 32,221,184 * 2 = 64,442,368 queries. Amazingly, NONE of them failed!
  • Valkey was used to store intermediate counts, running on the same instance as the node.js scripts.
    • Valkey memory usage: 1.82 GB (from valkey-cli info memory)
    • RDB backup file is only 241 MB.
    • 27,762,263 valkey HSET commands – none of them failed.
  • An instance with 32 RAM was used to be on the safe side. But it doesn't look like much memory gets used at all. According to the graphs, peak memory usage was just about 2.5 GB - this is almost same as the current memory usage because valkey is in-memory. Node.js appears to use a negligible amount of memory.
  • Disk usage: 552 MB of files to store user id lists from db.

The only hiccup was a bug in the replica views, T404473, which prevented the script from working for commons and testwiki, but it is possible to re-run for those specific wikis once it's fixed.

After that, the next step is to test-run the 2nd part of the script (make-global-list), but I'm in no rush as the original requirement T400228: SecurePoll for Wikimédia France board of administrators seat turns out to be not required any more as the election has only one candidate.

fgiunchedi subscribed.

Given the requirements I am wondering: have you considered running the code in Toolforge? There is a redis available, though you could also spin up your own in containers. Running in Toolforge will provide you with many benefits out of the box and save you from managing your own VMs.

Given the requirements I am wondering: have you considered running the code in Toolforge? There is a redis available, though you could also spin up your own in containers. Running in Toolforge will provide you with many benefits out of the box and save you from managing your own VMs.

There's interest in keeping the intermediate data – Toolforge redis installation or container installations don't provide a way to take RDB backups as far as I know. Also, I'm not sure if allocating GBs of memory in the toolforge k8s cluster works well. There's also a dependency on the filesystem for streaming large MariaDB resultsets, which is again shaky on Toolforge as we'd be relying on NFS.

I actually forgot to update this ticket with details of the full script run done after T404473 was fixed. I modified the algorithm somewhat so that the make-global-list step is simplified at the cost of using more Valkey memory.

Here are figures from the full test run (from October):

  • Valkey memory usage: 2.47 GB. The higher memory usage is because now it stores the counts against usernames instead of ids. This way, we skip the step of looking up names later. As only names are global, merging has to be based on names.
  • RDB file size: 984 MB
  • Disk usage (for temporary files): 715 MB
  • Runtime: ~3 hours. Seeing that s1 (enwiki) takes up substantially more time than s2-s8, I used a parallelism of 4 for querying s1, and 1 for s2-s8. The increased concurrency didn't cause any queries to time out or otherwise fail.