Page MenuHomePhabricator

WikiWho: pickles are too big for pickle_storage
Closed, ResolvedPublic

Description

Our first pickle_storage volume (that contains en, de, es, eu and tr) is 98% full. We need to move some languages to pickle_storage02 which is 57% full. I'm thinking the first volume can be more or less solely reserved for English, as that's the biggest and will grow the quickest.

After some of the non-English languages are moved to the 2nd volume, we should be safe for while, maybe… but overall I think it's probably time to re-invest in WikiWho and devise a more effective storage system than Python pickles.

As of 2025-11-01:

LanguageSize (GB)Article countLocation
English (en)~3,2007,083,607pickles
French (fr)7032,717,846pickles-02
German (de)7023,064,852pickles
Spanish (es)5352,071,519pickles
Italian (it)4551,942,311pickles-02
Japanese (ja)3601,478,586pickles-02
Polish (pl)2471,673,469pickles-02
Portuguese (pt)2401,158,983pickles-02
Dutch (nl)1892,200,912pickles-02
Arabic (ar)1741,284,342pickles-02
Turkish (tr)105650,610pickles
Hungarian (hu)97562,287pickles-02
Indonesian (id)90748,314pickles-02
Basque (eu)32474,423pickles

Event Timeline

MusikAnimal triaged this task as High priority.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
MusikAnimal renamed this task from pickle_storage is 98% full to pickle_storage is 99% full.Jan 7 2026, 11:43 PM

This must be done in the near future, or we will have a lot of unhappy users. Sorry I didn't find time to tend to it before my sabbatical :/

I left some notes at https://www.mediawiki.org/wiki/Talk:XTools#c-MusikAnimal-20260108234500-Lustiger_seth-20260107224600

Thanks so much in advance!

This may be better fit for the Community-Tech (Fox Squad), if @Ragesoss is unable to help. @TheresNoTime has worked on WikiWho before. Sam I'm sure can help too but looks like he needs to be added to the wikiwho VPS project https://openstack-browser.toolforge.org/project/wikiwho

I did a quick simple intervention today, deleting a portion of the pickles that hadn't been accessed in 90 days or more. Any that get requested will be regenerated, so it should only cause some minor inconvenience when the data isn't available on first request. (I ran a simple script that was looping through the en directory, and deleting each file if its atime was earlier than 90 days ago. I was disconnected with a broken pipe before it finished, so I don't know how much more it would have deleted it it ran to completion.)

Now we're 89%, so we should have some headroom for the near future.

Ragesoss renamed this task from pickle_storage is 99% full to WikiWho: pickles are too big for pickle_storage.Jan 10 2026, 3:00 AM

Hi @Ragesoss ! Is this something you are looking into? We are wondering if Community-Tech should take a look. Thank you for your help in advance :)

I tried a Soft Reboot Instance through https://horizon.wikimedia.org/project/ and that didn't seem to do anything.

Hi @Ragesoss ! Is this something you are looking into? We are wondering if Community-Tech should take a look. Thank you for your help in advance :)

I'm not planning to look into long-term solutions for the pickle storage problem, but I think the workaround I described in https://phabricator.wikimedia.org/T407660#11509396 can keep the system running for now. pickles is currently at 90%, and pickles-02 is at 58%. Deleting the ones that haven't been accessed in a long while feels like a fine compromise to me, as the system will regenerate the pickle for any article that gets requested. It just means a delay for the end-user while that article's history gets reprocessed.

I think just throwing some additional storage at it (like a third 5T volume), and reorganizing the storage as proposed in the task description — putting English on it's own volume — would let us kick the can down the road for a long while and enable all the languages.

We could also use that sort of delete-some-pickles strategy to add support for additional languages without more storage, if we don't pre-populate any pickles and simply rely on on-demand processing for requested articles. Then we could just use a cron job or similar to regularly delete pickles that haven't been accessed in X time period for all languages. That would perhaps be an unsatisfying kludge, but I think the alternative would be something approaching a complete rewrite of service.

@TheresNoTime I don't have much experience with pickle, do you think you take a look at this storage pickle issue?

just a snapshot of state:

root@wikiwho01:~$ df -h | grep /dev/sd
/dev/sda1        20G   18G  738M  97% /
/dev/sda15      124M   11M  114M   9% /boot/efi
/dev/sdc        4.9T  4.2T  423G  92% /pickles
/dev/sdb        4.9T  2.7T  2.0T  58% /pickles-02
root@wikiwho01:~# du -h /pickles
107G	/pickles/tr
4.0K	/pickles/dumps
16K	/pickles/lost+found
543G	/pickles/es
2.9T	/pickles/en
712G	/pickles/de
33G	/pickles/eu
4.2T	/pickles
root@wikiwho01:~# du -h /pickles-02/
462G	/pickles-02/it
181G	/pickles-02/ar
244G	/pickles-02/pt
93G	/pickles-02/id
192G	/pickles-02/nl
367G	/pickles-02/ja
77M	/pickles-02/dumps/it/logs
22G	/pickles-02/dumps/it
1.4M	/pickles-02/dumps/ar/logs
8.4G	/pickles-02/dumps/ar
4.8M	/pickles-02/dumps/pt/logs
11G	/pickles-02/dumps/pt
2.2M	/pickles-02/dumps/id/logs
3.0G	/pickles-02/dumps/id
1.9M	/pickles-02/dumps/nl/logs
9.5G	/pickles-02/dumps/nl
2.5M	/pickles-02/dumps/ja/logs
20G	/pickles-02/dumps/ja
11M	/pickles-02/dumps/pl/logs
13G	/pickles-02/dumps/pl
7.0M	/pickles-02/dumps/fr/logs
40G	/pickles-02/dumps/fr
1.5M	/pickles-02/dumps/hu/logs
4.6G	/pickles-02/dumps/hu
129G	/pickles-02/dumps
16K	/pickles-02/lost+found
251G	/pickles-02/pl
714G	/pickles-02/fr
99G	/pickles-02/hu
2.7T	/pickles-02/

Created the pickles-03 volume (5000GB), mounted at /pickles-03

(or, it will be when formatted)

root@wikiwho01:~# screen -ls
There is a screen on:
	2369842.wmcs-prepare-cinder-volume	(02/09/26 18:57:37)	(Detached)

I've been doing some cleaning up of pickle files which are over 4 years old in preparation for moving eu/es/de to pickles-02

Migrations

  • /pickles/eu migrated to /pickles-02/eu
  • /pickles/tr migrated to /pickles-02/tr
  • /pickles/es migrated to /pickles-02/es
  • /pickles/de migrated to /pickles-03/de

Running changes

diff --git a/wikiwho_api/settings_wmcloud.py b/wikiwho_api/settings_wmcloud.py
index 4f5d349..0aa96f2 100644
--- a/wikiwho_api/settings_wmcloud.py
+++ b/wikiwho_api/settings_wmcloud.py
@@ -23,14 +23,16 @@ ACTIONS_LANGUAGES = ['ar', 'de', 'en', 'es', 'eu', 'fr', 'hu', 'id', 'it', 'ja',
 EVENT_STREAM_WIKIS = ['arwiki', 'dewiki', 'enwiki', 'eswiki', 'euwiki', 'frwiki', 'huwiki', 'idwiki', 'itwiki', 'jawiki', 'nlwiki', 'plwiki', 'ptwiki', 'trwiki']

 # On pickle_storage volume, mounted to /pickles
-PICKLE_FOLDER_DE = '/pickles/de'
+#PICKLE_FOLDER_DE = '/pickles/de'
 PICKLE_FOLDER_EN = '/pickles/en'
-PICKLE_FOLDER_ES = '/pickles/es'
-PICKLE_FOLDER_EU = '/pickles/eu'
-PICKLE_FOLDER_TR = '/pickles/tr'
+#PICKLE_FOLDER_ES = '/pickles/es'
+#PICKLE_FOLDER_EU = '/pickles/eu'
+#PICKLE_FOLDER_TR = '/pickles/tr'

 # On pickle_storage02 volume, mounted to /pickles-02
 PICKLE_FOLDER_AR = '/pickles-02/ar'
+PICKLE_FOLDER_ES = '/pickles-02/es' # Moved from /pickles/eu @ 2026-02-09, T407660
+PICKLE_FOLDER_EU = '/pickles-02/eu' # Moved from /pickles/eu @ 2026-02-09, T407660
 PICKLE_FOLDER_FR = '/pickles-02/fr'
 PICKLE_FOLDER_HU = '/pickles-02/hu'
 PICKLE_FOLDER_ID = '/pickles-02/id'
@@ -39,6 +41,10 @@ PICKLE_FOLDER_JA = '/pickles-02/ja'
 PICKLE_FOLDER_NL = '/pickles-02/nl'
 PICKLE_FOLDER_PL = '/pickles-02/pl'
 PICKLE_FOLDER_PT = '/pickles-02/pt'
+PICKLE_FOLDER_TR = '/pickles-02/tr' # Moved from /pickles/tr @ 2026-02-09, T407660
+
+# On pickle_storage03
+PICKLE_FOLDER_DE = '/pickles-03/de' # Moved from /pickles/tr @ 2026-02-10, T407660

 REST_FRAMEWORK['DEFAULT_THROTTLE_RATES']['anon'] = '100/sec'
 REST_FRAMEWORK['DEFAULT_THROTTLE_RATES']['burst'] = '100/sec'

(should be reflected in https://github.com/wikimedia/wikiwho_api/pull/21 as we go)

Next

  • Deleting /pickles/de in a screen session

Deleting /pickles/de in a screen session

It seems this was done.

As far as I can tell this task can now be resolved. We've another 5TB volume now \o/

Thank you for you help, @TheresNoTime!