Page MenuHomePhabricator

worker_long queue is always stuck; Very large and revision-rich articles are unavailable
Open, LowPublic

Description

From https://wikiwho-flower.wmcloud.org/ (maintainer-only), I have deduced that very large and revision-rich articles like Barack Obama cannot feasibly be processed from scratch – that is, to process the entire revision history to create a new pickle storage file when one did not previously exist. It will always time out. This is why we used XML files for the initial import, both to reduce the work the algorithm has to do, and to avoid having to query the action API en masse. For pages with an existing pickle, this is not a problem, as we only need to append to the storage as new revisions are made, and don't have to reprocess the entire revision history.

In testing T414075, I decided the delete the pickle file for [[Barack Obama]], and so now that page is indefinitely unusable in WikiWho.

To make matters worse, new requests come in and hog the queue when trying to rebuild the pickle file, and I think that's more or less our issue with the queue going offline. The initial request will be under the worker_user queue, and you can search for it in Flower. That will eventually time out, then it gets moved to worker_long, where it will time out again after an hour or so.

The new global API limitations at Wikimedia also may be at play here (though Cloud Services is meant to be exempt). At any rate, it's clear we need to do something. Quite a few other articles are unavailable in WikiWho for the same reason. It stands to reason for Wikipedias that high-revision mainspace articles are probably of great interest, so ideally Who-Wrote-That, Education-Program-Dashboard, XTools and other integrations will be able to support said articles.

Event Timeline

I decided the delete the pickle file for [[Barack Obama]], and so now that page is indefinitely unusable in WikiWho.

I tried to run the generate_articles_from_wp_xmls locally from a 7z-compressed XML export built with Special:Export, with intention to just SCP it over the server, but my attempt errored out with:

2026-04-03 05:10:00,669 ForkProcess-1 Barack_Obama ERROR    Barack Obama-(534366)--------#######*******#######
Traceback (most recent call last):
  File "/home/musikanimal/dev/wikiwho_api/wikiwho/management/commands/generate_articles_from_wp_xmls.py", line 57, in generate_articles
    page_ = sorted(list(page), key=operator.attrgetter('timestamp'))
  File "/home/musikanimal/dev/wikiwho_api/env/lib/python3.9/site-packages/mwxml/iteration/page.py", line 32, in __iter__
    for revision in self.__revisions:
  File "/home/musikanimal/dev/wikiwho_api/env/lib/python3.9/site-packages/mwxml/iteration/page.py", line 44, in load_revisions
    yield Revision.from_element(first_revision)
  File "/home/musikanimal/dev/wikiwho_api/env/lib/python3.9/site-packages/mwxml/iteration/revision.py", line 61, in from_element
    raise MalformedXML("Unexpected tag found when processing " +
mwxml.errors.MalformedXML: Unexpected tag found when processing a <revision>: 'origin'

I'll pick up from here tomorrow and at least get this example back so we can close out T414075.

MusikAnimal changed the task status from Open to In Progress.Apr 3 2026, 5:24 AM

Updates: For [[Barack Obama]], specifically, I realized using Special:Export both adds the unwanted <origin> tag, and also it is limited to a maximum of 1,000 revisions. Instead, I tediously went through the raw dumps to find the one containing [[Barack Obama]], and edited the ~17GB XML file to include only the desired page (which is still 5GB by itself!). Now I'm trying to import that, but keep running into timeouts. I've increased the timeout threshold and am trying again. This could take hours, it seems!!

As for resolving the issue with the worker_long queue going offline, my hope is that after we generate the pickle files for these very large articles and copy them over to production, the queue won't get stuck as it only needs to append to the existing pickle rather than create it from scratch. We'll likely want to make a script to automate this process.

MusikAnimal lowered the priority of this task from Medium to Low.Apr 10 2026, 6:08 AM

Alright, [[Barack Obama]]'s pickle is back, hehe :) In my testing, it takes about the same amount of time to load as before T414075, maybe slightly faster, but certainly not slower.

There's still more work to do here, but I'm going to de-prioritize as the larger effort will likely be very time-consuming. I just wanted to document the issue I discovered, and get our benchmark article back in production.

MusikAnimal changed the task status from In Progress to Open.Jun 30 2026, 6:55 PM
MusikAnimal removed MusikAnimal as the assignee of this task.