Page MenuHomePhabricator

Need to free up space in both Discourse instances
Closed, ResolvedPublic

Description

Both instances at https://tools.wmflabs.org/openstack-browser/project/discourse need more free space in order to upgrade Discourse. https://discourse.wmflabs.org/ has ~3GB free and cannot be upgraded regularly anymore. https://discourse-mediawiki.wmflabs.org/ has ~5GB free and could still be upgraded, running the option to clean Docker containers.

There is something fishy here. Discourse and our content & backups don't require the space that is being taken, not at all. After a quick inspection, both instances seem to have growing Docker directories under /var/lib. discourse1002.discourse.eqiad.wmflabs also has multiple GBs in directories for devicemapper and overlay that I believe we are not using anymore.

Searching around I have found pages about Docker folders growing endlessly and what to do with them, but I don't dare to touch anything because my Docker knowledge is very limited, and that is already an overstatement. I don't even know whether the old devicemapper and overlay directories can be simply rm or they should be purged with some Docker instruction.

Please help. :)

Event Timeline

Qgil triaged this task as High priority.Jan 2 2019, 8:43 AM
Qgil created this task.

Just stop the app, delete all containers and rebuild it? (The Docker command is docker rm / docker rmi; I don't know if deleting files by hand is safe.) In theory there's a ./launcher cleanup command to automate it.

./launcher cleanup worked well for discourse-mw and now we have 10 GB free there.

discourse1002, however... it seems that we cannot reclaim more space via Discourse's launcher app or Docker commands.

There we had to change the storage driver. Currently Docker handles the overlay2 storage, and devicemapper and overlay don't seem to "be seen" by Docker anymore. There we have about 5.5GB used for apparently no good reason, and I don't know how to remove them "cleanly" (other than removing the directories manually):

3.6GiB [##########] /overlay2
3.1GiB [########  ] /overlay
2.4GiB [######    ] /devicemapper
Qgil claimed this task.

I tried this:

  1. Stop discourse app
  2. Rename /devicemapper directory
  3. Start discourse app
  4. Check that everything works fine (and everything was fine indeed)
  5. Delete /devicemapper and all its contents.

After this, I had enough free space to rebuild the app. Good enough for now?

Thanks @Qgil! :) I'd wondered what the issue was there. Sounds like a good fix.