I'm seeing a lot of No space left on device errors, and

root@prod0:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        20G   20G     0 100% /

which I'm guessing isn't going to help anything much.

looks like our project nfs mount went away so writes were landing on the primary volume instead.

also, our docker data dir has gotten pretty heavy. We keep that on a separate cinder volume for our newer projects. I'll just start the vm migration process that needs to happen anyway. as it's much easier to just implement that on a fresh vm than to do the move in place.

okay, I've got to move to new docker data volume in place; things will be down for a bit.

storage is now happy; I brought things back up to have a look and the sessions table is a bit of a mess; turning off web service for a bit so I can do some house keeping

okay, I extracted the session table from the last good backup, dropped the current table (which was corrupted), and restore is underway.

Still down for me

In T430155#12055952, @Finnfrog99 wrote:

Still down for me

This is expected while we are restoring. I'll give a status update when everything is back up.

In greater context....

Okay, the table is cleaned up. Validating that I can write a good backup before bringing the site up.

Verified. Our sessions table grew quite large. I'm pruning it as well using django manage. This has not been the fastest way to restore the service, but it's pretty foolproof, which is the best way to come in after a long time away from the project. I appreciate everyone's patience.

Hello @jsn.sherman! I have not yet started work on this, but twl is one of the projects that needs an NFS update sometime soon (see T429793). Do you have any idea what went wrong with the NFS mount? And, is this a project that still requires nfs for sharing files between hosts?

don't know what happened, but I think it was an ephemeral issue. Things are mounted up properly now. but we're pitching NFS overboard as part of T402054: twl: Replace deprecated Bullseye VMs in Cloud VPS this sprint. This issue just happened to pop up first. I actually have an item to ask you if there's anything special we need to do to decommission NFS, other than dropping it from any heira configs and killing the related instance/volumes.

Session pruning is still ongoing; either our daily pruning job was not working or we got a large bump in session creation and didn't notice. I'll do an analysis of the last good session dump before this manual pruning and also look at the history for the job. That will be for another day, but I will share more info in a comment here before I close the task out.

In the mean time, we'll be in better shape since we now have the root fs, the backup dir, and the docker data dir (which contains the db) all filling separate volumes.

okay, I didn't finish the pruning because we've been down so long. I've brought the site back up with a smaller-than-before session table. The management command is very safe, but also pretty slow. I'll take another look at it tomorrow. I may just see if I can do a good offline finishing job when it's time to migrate to the new vm.

we're pitching NFS overboard as part of T402054: twl: Replace deprecated Bullseye VMs in Cloud VPS this sprint.

That is great news for me! I will check twl off my list and leave you to it. Thanks for the quick reply!

Thanks all for the great work so far!

hmm, it's still looking swampy on the db side; I'm seeing db-driven timeouts and high io wait on the host. Investigating.

traffic isn't over the top and the only unhappy db logs are from the crash when the disk filled. Performing a ceremonial reboot.

I see now; the db is trying to do a transaction rollback against a few million rows on startup. This was not making it to the logging console before. I'm going to briefly take the site back down, clear out the db, and restore from backup. That way we won't have any bin log to try to reconcile.

the behavior of the restore is looking good; high user cpu time, moderate io/wait time. It does appear to be writing more slowly to the cinder volume than it was to the root volume, but I don't think that's going to be a big problem for this site. It does mean that we're twiddling our thumbs for a bit longer during this restore though.

This has been going on quite a long time and this restore is not done yet. It definitely should be!
I must call it a day, but the next step will be restoring on another host as a sanity check.

If it works fine on another host, we'll just skip straight to the vm replacement. If it doesn't, we'll investigate further.

Checking back in this morning and the db seems okay? I don't know if these cinder volumes are sparsely allocated under the hood and have slow "cold" performance or if it was just a blip? The restore did complete and I just manually cleared the session table (which had over 4m rows) for expediency. It looks like schedule jobs haven't run since January 30 based on the django cron entries, but we have backups (which are fired by that mechanism), so I don' t believe it. Looking at the commit log, I did make changes on that day https://github.com/WikipediaLibrary/TWLight/commits/master/?since=2026-01-30&until=2026-01-30 so most likely something there caused the django cron logs to quietly stop working.

I'm leaving the site up and running since it seems stable and we'll keep an eye on the next backup window: does the backup get created? does the job get logged correctly?

In T430155#12058908, @jsn.sherman wrote:

Checking back in this morning and the db seems okay? I don't know if these cinder volumes are sparsely allocated under the hood and have slow "cold" performance or if it was just a blip? The restore did complete and I just manually cleared the session table (which had over 4m rows) for expediency. It looks like schedule jobs haven't run since January 30 based on the django cron entries, but we have backups (which are fired by that mechanism), so I don' t believe it. Looking at the commit log, I did make changes on that day https://github.com/WikipediaLibrary/TWLight/commits/master/?since=2026-01-30&until=2026-01-30 so most likely something there caused the django cron logs to quietly stop working.

I'm leaving the site up and running since it seems stable and we'll keep an eye on the next backup window: does the backup get created? does the job get logged correctly?

It's back! thank you so much. You might want to check your enwiki talkpage

In T430155#12059372, @Finnfrog99 wrote:

In T430155#12058908, @jsn.sherman wrote:

Checking back in this morning and the db seems okay? I don't know if these cinder volumes are sparsely allocated under the hood and have slow "cold" performance or if it was just a blip? The restore did complete and I just manually cleared the session table (which had over 4m rows) for expediency. It looks like schedule jobs haven't run since January 30 based on the django cron entries, but we have backups (which are fired by that mechanism), so I don' t believe it. Looking at the commit log, I did make changes on that day https://github.com/WikipediaLibrary/TWLight/commits/master/?since=2026-01-30&until=2026-01-30 so most likely something there caused the django cron logs to quietly stop working.

I'm leaving the site up and running since it seems stable and we'll keep an eye on the next backup window: does the backup get created? does the job get logged correctly?

It's back! thank you so much. You might want to check your enwiki talkpage

You're gonna make me blush! Thanks for the recognition 💛