Notes for me:

• https://www.mediawiki.org/wiki/Moderator_Tools/Development/Cloud_VPS
• https://www.mediawiki.org/wiki/Moderator_Tools/Development/Cloud_VPS/Hashtags
• https://github.com/WikipediaLibrary/TWLight/wiki/Debian-Server-setup

PR:
https://github.com/WikipediaLibrary/TWLight/pull/1541

@taavi I did a dry run on replacement using our new rootless docker setup and found kernel.unprivileged_userns_clone to be very much not-enableable. I was able to flip that on when I migrated the hashtags vm to this configuration during its trixie upgrade (T402056). It looks flipped back off there as well now too, making me think that we should not reboot that box (currently up for 218 days). Is it a reasonable ask to be able to enable it in order to run our docker compose managed services as not-root?

Unprivileged user namespaces are disabled on Wikimedia servers by default since on the vast majority of cases, it's primary function on single-tenant server machines is facilitating kernel local privilege escalation vulnerabilities. You do have the option of turning it back on in Hiera [0]. (I personally don't see a lot of benefit of rootless Docker in server environments - the user you're managing the server and the services as probably already has the ability to become root, so as long as you run the processes in the containers as a non-root user you ihmo get a very similar level of security for the containers without the additional attack surface of unprivileged userns's.)

[0]: profile::base::unprivileged_userns_clone: true

In T402054#12104493, @taavi wrote:

Unprivileged user namespaces are disabled on Wikimedia servers by default since on the vast majority of cases, it's primary function on single-tenant server machines is facilitating kernel local privilege escalation vulnerabilities. You do have the option of turning it back on in Hiera [0]. (I personally don't see a lot of benefit of rootless Docker in server environments - the user you're managing the server and the services as probably already has the ability to become root, so as long as you run the processes in the containers as a non-root user you ihmo get a very similar level of security for the containers without the additional attack surface of unprivileged userns's.)

[0]: profile::base::unprivileged_userns_clone: true

TIL! Thanks @taavi! FWIW, we use an unprivileged account that we all su into for managing things.

The deploy commit is merged and CI is green: the production branch carries the rootless-compose deployment and quay has the branch_production image. What's left here is the cutover itself, which I'm running in one recorded sitting on a fresh Trixie VM provisioned by the toolkit from T432043: Consolidate rootless docker compose VM deployment. Old swarm prod0 keeps serving until the final proxy flip, so rollback is a flip back.

This also retired twl-nfs-1: backups moved onto a dedicated cinder volume mounted at /usr/local/backup, so the new VM carries no NFS dependency.

The steps in the description predates the toolkit (it points at the old per-environment compose files and the NFS-to-cinder copy); the checklist below replaces it.

Cutover

Confirm before starting

• branch_production tag is present and current in quay
• space for two cinder volumes, 30 GiB each: one for docker data (holds the mariadb volume under docker's data-root), one for backups (30/30 retention in production)
• Media is ~8 MiB of thumbnails after ten years, so the root-fs bind at /srv/TWLight/media is a non-issue. Noted only because a full root fs was the earlier outage (T430155: The Wikipedia Library is down); this is nowhere near it
• Backup tarball format: gzipped tar carrying ./twlight.sql and ./media; restore.sh aborts if either is missing before it touches the database

1. Create storage and launch the VM (horizon)

• Create two 30 GiB volumes: docker data, backups
• Launch a Debian Trixie instance, flavor sized for the full stack (current image is g4.cores4.ram8.disk20), security group allowing 80/443/22
• Instance metadata under meta:
  • git_remote: https://github.com/WikipediaLibrary/TWLight.git
  • env: production
• Cloud-init user-data (the only bootstrap; everything else derives from metadata):

#!/usr/bin/env bash
set -eo pipefail
apt-get update && apt-get install -y git
git clone https://gitlab.wikimedia.org/repos/modtools/cloudvps-compose-deploy.git \
  /opt/cloudvps-compose-deploy
exec /opt/cloudvps-compose-deploy/bin/init.sh

• Attach both volumes once the instance is up, note the device ids in "attached to"
• Wait for cloud-init, peeking at the full console log in horizon (https://horizon.wikimedia.org/project/instances/<instance_id>/console) for a clean init.sh finish

init.sh (root, from cloud-init) adds swap, installs docker and the rootless extras, disables the root daemon, creates the lowercased twlight account, clones to /srv/TWLight and checks out production, symlinks the toolkit daemon.json, seeds .env from template.env.

2. Prepare volumes and finish provisioning (SSH, sudo login user)

• Prepare each cinder volume (once per volume, one to each mountpoint):

sudo wmcs-prepare-cinder-volume   # docker data -> /usr/local/docker-data
sudo wmcs-prepare-cinder-volume   # backup      -> /usr/local/backup

• Run the second half:

sudo /opt/cloudvps-compose-deploy/bin/post-init.sh

It chowns the mounted volume roots to twlight, enables lingering, then re-execs as twlight to install the rootless daemon and crontab. In production it refuses to continue unless both mountpoints sit on their own block device, so a missed volume prep fails loud here rather than filling the root fs later.

3. Bring the stack up on empty data (SSH as twlight)

sudo su twlight (lands in /srv/TWLight with the docker context and COMPOSE_FILE set)

The */5 deploy cron starts the stack within five minutes on its own. temporarily disable for first time deploy. eg. crontab -e and comment everything out
and:

docker compose up -d
docker compose ps
curl -sI -H 'Host: wikipedialibrary.wmflabs.org' http://localhost/

DB is empty here; a 301 proves nginx is up, not that data is present. Old prod0 is still the live site.

4. Move the data (the data drift window)

• Window start
  • comment out crontab
  • fresh backup tarball on old prod, and confirm both members: tar -tzf archive.tar.gz ./twlight.sql ./media >/dev/null && echo ok
• Copy it to the new VM's backup volume: scp archive.tar.gz new:/usr/local/backup/
• Restore inside the app container (prechecks members, extracts media, drop-and-recreate SQL import, reowns media, migrates):

docker compose exec -T twlight bin/restore.sh backup/archive.tar.gz

Success ends with "Finished TWLight restore." Stopping after "Finished importing" without that line means the media chown or migrate step failed; check before flipping.

5. Go live (horizon)

• DNS -> Web Proxies: edit the wikipedialibrary.wmflabs.org proxy to target the new instance at port 80
• [x Watch the new VM as traffic arrives: docker compose logs -f twlight, plus Glitchtip
• End-to-end over the real hostname: OAuth login round trip, a few authenticated pages, media loads, admin

If anything looks wrong, flip the proxy back to old prod0; it is untouched and still authoritative.

6. QA

• deploy.sh tick logs Up to date or a redeploy, and the marker under ~/.cloudvps-compose-deploy/ matches
• Backup cron produces an archive on /usr/local/backup and prunes to 30/30
• runcrons tick runs clean
• One deliberate reboot: the rootless daemon comes back (lingering) and the volumes remount from fstab

Rollback

• Before the proxy flip: old prod0 never stopped serving; abandon or fix the new VM, no user impact
• After the flip: point the proxy back at prod0; re-enable crons
• Keep prod0 through the soak before deleting it, then delete prod0

Follow-ups (not this sitting)

• Delete prod0 after the soak
• Managed-DB evaluation so the DB survives the next VM replacement (T432124: Evaluate Trove for the production database)

note that I tested email sending on the new prod and it worked as expected with wmf and non wmf addresses. I threw away the staging dkim key material during testing. staging emails to console by default, so there's no disruption here, but we will need to go through the dkim + dns setup for staging if we want to re-enable in the future.
https://github.com/WikipediaLibrary/TWLight/wiki/Debian-Server-setup#dkim

leaving open until next week to discover any latent issues.

looking good, but recommend deferring deletion of the old vm until we've had 1 clear week of deploys.