The wikilink production instance (prod.wikilink.eqiad1.wikimedia.cloud) is a Debian Bullseye VM slated for retirement under the 2025 Cloud VPS Bullseye deprecation, deadline 2026-07-31. I'm replacing it with a fresh Debian Trixie VM and, in the same move, retiring Docker Swarm for the rootless docker compose deployment from the CloudVPS Compose Deploy toolkit, the same cutover hashtags (T402056) and TWLight (T402054) already ran. The deploy environment is the master branch.

wikilink is a reporting tool, so downtime during the cutover is fine. Two wikilink specifics still change the steps:

• It ingests a live event stream. linkevents_collect writes LinkEvent rows continuously, so a point-in-time DB copy leaves a gap. After the restore, linkevents_collect --historical replays from the restored high-water mark and closes it.
• The data move is DB-only. The source is bind-mounted from the git checkout (the image carries only the runtime and its dependencies), and the only docker volume is the MariaDB data. So the new VM gets a fresh toolkit-prepared docker-data volume and the DB comes over as a single gzipped mysqldump. Old prod keeps its volumes untouched for rollback.

Prerequisite: the rootless-compose deploy config on master

CI already publishes the branch_master images (externallinks, eventstream, externallinks_cron), and the base compose already bind-mounts the checkout, so the overlay is small: a docker-compose.deploy.yml carrying the deploy-only bits (pull instead of build, HOST_BACKUP_DIR, no dev port publishes), template.env aligned to the toolkit contract, and the crontab moved to conf/crontab with the deploy tick.

Checklist

1. Create storage and launch the VM (horizon)

• Create two cinder volumes: docker data, backup
• Launch a Debian Trixie instance, a flavor sized for the full stack, security group allowing 80/443/22
• Instance metadata under meta:
  • git_remote: https://github.com/WikipediaLibrary/externallinks.git
  • env: master
• Cloud-init user-data (the only bootstrap; everything else derives from metadata):

#!/usr/bin/env bash
set -eo pipefail
apt-get update && apt-get install -y git
git clone https://gitlab.wikimedia.org/repos/modtools/cloudvps-compose-deploy.git \
  /opt/cloudvps-compose-deploy
exec /opt/cloudvps-compose-deploy/bin/init.sh

• Attach both volumes once the instance is up
• Watch the console log in horizon for a clean init.sh finish

init.sh (root, from cloud-init) adds swap, installs docker and the rootless extras, disables the root daemon, creates the lowercased externallinks account, clones to /srv/externallinks and checks out master, symlinks the toolkit daemon.json, seeds .env from template.env.

2. Prepare volumes and finish provisioning (SSH, sudo login user)

• Prepare each cinder volume, once per volume:

sudo wmcs-prepare-cinder-volume   # docker data -> /usr/local/docker-data
sudo wmcs-prepare-cinder-volume   # backup      -> /usr/local/backup

• Run the second half:

sudo /opt/cloudvps-compose-deploy/bin/post-init.sh

It chowns the mounted volume roots to externallinks, enables lingering, then re-execs as externallinks to install the rootless daemon and crontab. In production it refuses to continue unless both mountpoints sit on their own block device, so a missed volume prep fails loud here rather than filling the root fs later.

3. Bring the stack up on an empty DB (SSH as externallinks)

sudo su externallinks lands in /srv/externallinks with the docker context and COMPOSE_FILE set. Comment out the deploy cron for the manual bring-up (crontab -e), then bring up only what the restore needs, holding eventstream and crons so nothing writes to the empty DB:

docker compose up -d db externallinks
docker compose ps

4. Maintenance window: move the data

• On old prod, quiesce the writers (stop its eventstream and cron services) so the dump is consistent, then take a fresh backup (backup.py) to its backup volume
• Copy the dump to the new VM's backup volume:

scp old:/usr/local/backup/YYYYMMDD.sql.gz new:/usr/local/backup/

• Restore inside the app container on the new VM, on a TTY, answering y at the drop prompt (restore.sh is interactive and reads stdin, so no -T):

docker compose exec externallinks bin/restore.sh backup/YYYYMMDD.sql.gz

• It drops and reimports the DB, then migrates; success ends with "Finished restore."
• Bring up the rest of the stack; eventstream starts with --historical and backfills from the restored high-water mark:

docker compose up -d
docker compose logs -f eventstream   # confirm it catches up to live

5. Go live (horizon)

• Web Proxies: point the wikilink.wmflabs.org proxy at the new instance on port 80
• Re-enable the deploy cron on the new VM (crontab -e)
• Verify over the real hostname: the homepage, an organisation page, a collection page with its graphs, and the admin. Watch docker compose logs -f externallinks eventstream plus Sentry

6. QA

• deploy.sh tick logs up-to-date or a redeploy, and the toolkit state marker matches
• Backup cron writes a dump to /usr/local/backup and prunes to 14 days
• The cron batch runs clean (aggregate fills, linksearchtotal_collect, linkevents_archive, users_update_lists)
• eventstream stays connected and the LinkEvent high-water mark tracks live
• One deliberate reboot: the rootless daemon comes back (lingering), the volumes remount from fstab, and eventstream reconnects

Rollback

• Before the proxy flip: old prod is untouched; abandon or fix the new VM and restart old's writers, no lasting impact
• After the flip: point the proxy back at old prod, restart its stack; its consumer resumes via --historical from its last stored event (within the stream's replay retention), so the LinkEvent record closes back up
• Keep old prod through the soak, then delete the instance and its cinder volumes

Acceptance criteria

Deploy config

• docker-compose.deploy.yml, conf/crontab, and template.env conform to the toolkit contract; a bare docker compose up still serves local dev through the override slot
• a deploy on master (git pull plus compose up from the toolkit) brings the full stack up from the branch_master images

Cutover

• new Trixie VM provisioned by the toolkit with prepared cinder volumes for docker data and backup
• DB restored from a maintenance-window backup; restore.sh finishes clean
• event stream backfilled so there is no LinkEvent gap between the backup and go-live
• proxy flipped to the new VM; the site loads over wikilink.wmflabs.org, the organisation, collection, and program pages render their graphs, and admin works

QA

• deploy cron ticks clean and the toolkit state marker matches
• backup cron writes a dump to /usr/local/backup and prunes to the 14-day retention
• the cron batch runs clean (aggregate fills, linksearchtotal_collect, linkevents_archive, users_update_lists)
• eventstream stays connected and the LinkEvent high-water mark tracks live
• one deliberate reboot brings the rootless daemon, the volume mounts, and eventstream back

Decommission

• old prod.wikilink kept through the soak, then shut down and deleted, its cinder volumes deleted