The cloud-vps 'cvn' project needs a new NFS server. The data will be preserved, but there will be a bit of downtime during the switch and all client hosts will need to be rebooted.
Admins, please let me know if I need to coordinate this with others or if I can just make the switch whenever.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T429793 Update all cloud-vps Bullseye NFS servers | |||
| Resolved | Andrew | T430589 CVN: Replace Bullseye nfs server with a new Trixie server |
I'd prefer not having to think about what the processes are going to do with NFS absent. So maybe a clean shutdown first, and then a clean wakeup after would be better.
But, let me look at how/where we're using it. If I can rule out CVNBot's running code, the rest are simple shell scripts that I can reason about.
How it should work:
Once confirmed, I think we can just do a live change as you suggest, and then fixup/reboot.
https://codesearch.wmcloud.org/search/?q=data%2Fproject%2Fcvn&files=&excludeFiles=&repos=
The three places where it's used are cronjobs. (There is a fourth reference tasks.yaml which is a runbook for manual sysadmin maintenance.)
#!/bin/bash -eu … tar -cvzpf "/data/project/cvn-common/backup/$host-services.tar.gz" /srv/cvn/services
#!/bin/bash -eu … LOCAL_DIR=/srv/cvn/services/cvnbot/CVNBot14 NFS_DIR=/data/project/cvn-common/dropbox # Move to NFS mv "$LOCAL_DIR/.cp.$NFS_FILENAME" "$NFS_DIR/.cp.$NFS_FILENAME"
#!/bin/bash -eu … LOCAL_DIR=/srv/cvn/git/cvn-api/data NFS_DIR=/data/project/cvn-common/dropbox # Copy from NFS cp "$NFS_DIR/$NFS_FILENAME" "$LOCAL_DIR/.cp.$NFS_FILENAME"
@Andrew So long as these commands cleanly error or timeout (i.e. not exit zero and have bash execute the next line), then everything else is fine. Go ahead any time.
These cronjobs are okay to be down or fail for a few hours. Even if they end up partially overwriting a file, it is fine because the scripts use temp files to prevent the next consumer from reading incomplete data. And if that fails too somehow, then we're still fine because this data is not read in a way that can cause cascading effects (it just means the CVN Web API may not return any results for a while). Either way, the next run should clean it up soon enough.
Typically I would restart nfs clients after an nfs migration to make sure they're mounting the new server properly. So the most straightforward thing is for me to just shut down all the -app hosts in the cluster, flip the nfs switch, and then start them up again. That's also the best way to avoid any weird in-between states; does that sound ok? I'm assuming the above jobs run on the app hosts and not the apache host, please lmk if that's wrong.
@Andrew It runs on the Apache host as well. The backup and push tasks are for the app hosts. The pull task is for the apache host.
I'd prefer to keep the hosts running as long as possible, because the CVNBot app conveys vandal edits to patrollers using IRC, and the CVN API apache highlights suspicious edits to patrollers using Special:RecentChanges on-wiki (via a gadget).
The NFS tasks are not in the critical path and okay to fail for a while.
The push/pull tasks periodically sync the read-only web API with a bot app. The backup task is for backups.
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:24:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:24:54Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589)
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:26:11Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:26:16Z] <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.nfs.migrate_service (exit_code=97) (T430589)
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:26:44Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:30:25Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589)
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:30:38Z] <andrew@cloudcumin1001> START - Cookbook wmcs.nfs.migrate_service (T430589)
Mentioned in SAL (#wikimedia-cloud-feed) [2026-07-16T17:30:39Z] <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) (T430589)
This is now done. I did wind up doing a staggered reboot of all servers because NFS clients are not great at handling change.
The old nfs server (which contains no data but does contain a bit of config state) is now in a shutoff state. Once you're satisfied that things are working properly please delete that VM, close this task, and then upgrade the two Bullseye servers in that project to Trixie T395164
Thanks!
Mentioned in SAL (#countervandalism) [2026-07-16T20:04:12Z] <Krinkle> Delete cvn-nfs1 per T430589