Page MenuHomePhabricator

notebook1004 - /srv is full
Closed, ResolvedPublic


Icinga alerted that /srv on notebook1004 is full (100%, 1.1G left).

/srv/ includes home directories in /srv/home

The entire size is 128G and here are some users using more than 1G of it, ordered by size:

47G ryanmax
29G nathante
6.3G bearloga
2.2G ebernhardson
1.8G nettrom
1.6G otto

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Give that this is likely to impact other users can we temporarily compress that directory ( (/home/ryanmax) to make up space?

I deleted a little bit from my home dir, so we have a little bit of room for a bit. I'll give them a little time to respond.

I just deleted some files and I'm compressing others. I didn't realize space was so tight ... my apologies.

The Jupyter Notebook servers are meant mostly to be an GUI/Cli interface to Hadoop based systems. If you can, please consider storing data in HDFS.

@RyanSteinberg + 1 to andrew's suggestion. data should not be kept on notebook servers, rather you can keep it on your user database in hadoop. This is due to space concerns in notebooks but also it ensures your data would not be lost should the notebook machine fail entirely, data is replicated on hadoop but NOT on notebooks.

jbond triaged this task as Medium priority.Sep 9 2019, 9:43 AM
Ottomata claimed this task.

I deleted a couple Gb that I don't need. Unfortunately most of the space I'm using is from ORES assets so I can't really store it in Hadoop. Maybe I should move this work to a different machine with more space? It's a bit inconvenient to have to move between different machines for different tasks though. Is my current usage OK for the month or so? After that I'll clean up.

@Groceryheist Can you explain a bit why ORES assets cannot be stored in hadoop?

It looks like we have about 46G available for now, so hopefully that can
hold us over. If you don't mind, just keep an eye on usage, and if it gets
close to full, delete things:

df -h /dev/mapper/notebook1004--vg-data
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/notebook1004--vg-data  136G   84G   46G  65% /srv

Hey @Ottomata,
it turns out that I think stat1006 is a better fit for my purposes since it has ORES dependencies (mainly hunspell) that were missing on the notebook machine. So once I finish moving that part of the project over to stat1006 I can reduce my usage on notebook1004.

jijiki added a subscriber: jijiki.

Host alerted again about /srv being full, /srv/home is 119G.

@Groceryheist hello, can you check your home directory size ? :)

Mentioned in SAL (#wikimedia-operations) [2020-02-25T23:30:35Z] <mutante> notebook1004 - disk full once again (T232068)

We added SWAP to all stat100x hosts, and set a deprecation of the notebooks for June 2020. In theory we shouldn't receive any more alarms, closing.