Icinga alerted that /srv on notebook1004 is full (100%, 1.1G left).
https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=notebook1004&service=Disk+space
/srv/ includes home directories in /srv/home
The entire size is 128G and here are some users using more than 1G of it, ordered by size:
47G ryanmax
29G nathante
6.3G bearloga
2.2G ebernhardson
1.8G nettrom
1.6G otto
Give that this is likely to impact other users can we temporarily compress that directory ( (/home/ryanmax) to make up space?
I deleted a little bit from my home dir, so we have a little bit of room for a bit. I'll give them a little time to respond.
I just deleted some files and I'm compressing others. I didn't realize space was so tight ... my apologies.
The Jupyter Notebook servers are meant mostly to be an GUI/Cli interface to Hadoop based systems. If you can, please consider storing data in HDFS.
@RyanSteinberg + 1 to andrew's suggestion. data should not be kept on notebook servers, rather you can keep it on your user database in hadoop. This is due to space concerns in notebooks but also it ensures your data would not be lost should the notebook machine fail entirely, data is replicated on hadoop but NOT on notebooks.
I deleted a couple Gb that I don't need. Unfortunately most of the space I'm using is from ORES assets so I can't really store it in Hadoop. Maybe I should move this work to a different machine with more space? It's a bit inconvenient to have to move between different machines for different tasks though. Is my current usage OK for the month or so? After that I'll clean up.
It looks like we have about 46G available for now, so hopefully that can
hold us over. If you don't mind, just keep an eye on usage, and if it gets
close to full, delete things:
df -h /dev/mapper/notebook1004--vg-data Filesystem Size Used Avail Use% Mounted on /dev/mapper/notebook1004--vg-data 136G 84G 46G 65% /srv
Hey @Ottomata,
it turns out that I think stat1006 is a better fit for my purposes since it has ORES dependencies (mainly hunspell) that were missing on the notebook machine. So once I finish moving that part of the project over to stat1006 I can reduce my usage on notebook1004.
Mentioned in SAL (#wikimedia-operations) [2020-02-25T23:30:35Z] <mutante> notebook1004 - disk full once again (T232068)
We added SWAP to all stat100x hosts, and set a deprecation of the notebooks for June 2020. In theory we shouldn't receive any more alarms, closing.