Page MenuHomePhabricator

re-create codesearch instance / upgrade codesearch to trixie
Closed, ResolvedPublic

Description

I made a wrong click by accident that deleted the instance behind https://codesearch.wmcloud.org/

Now it's gone. There might be a chance to get it back from backups but it's not simple.

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Instance_backups#Restoring

Trying to recreate it (it was on bookworm, T429496) so either that or try on trixie right away.

Event Timeline

Mentioned in SAL (#wikimedia-cloud) [2026-06-22T19:29:51Z] <mutante> deleted instance codesearch9 by accident :( - creating instance codesearch10 on trixie - T429828 T429496

Mentioned in SAL (#wikimedia-cloud) [2026-06-22T19:56:44Z] <mutante> creating instance codesearch11 on bookworm - hound services failed to start on trixie - verify if it's distro related - T429828 T429496

Change #1304895 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] codesearch: ensure docker-cli is installed if on trixie or newer

https://gerrit.wikimedia.org/r/1304895

Change #1304895 merged by Dzahn:

[operations/puppet@production] codesearch: ensure docker-cli is installed if on trixie or newer

https://gerrit.wikimedia.org/r/1304895

Mentioned in SAL (#wikimedia-cloud) [2026-06-22T20:57:58Z] <mutante> - re-created on trixie! oof:)) should be working - https://codesearch.wmcloud.org/search/ T429828

Luckily, I managed to get it back up and working (and on trixie, so free upgrade).

https://codesearch.wmcloud.org/

Dzahn renamed this task from re-create codesearch instance to re-create codesearch instance / upgrade codesearch to trixie.Jun 22 2026, 9:06 PM

what was needed to fix it, in addition to puppet:

  • get Hiera data on instance level out of gerrit repo that shows Hiera edit history (not much, but should have it)
  • re-create web proxy codesearch.wmcloud.org and point it to port 3002 on new instance codesearch10
  • mount /dev/sdb /srv (first was not sure what path but it says so in a README in the repo and got lucky this used network storage volume that was still there)
  • ensure docker-cli is installed because that's a separate package on trixie (patch above)
  • systemctl daemon-reload and wait a while
Dzahn triaged this task as High priority.

Since we don't want to use /usr/local/sbin/wmcs-prepare-cinder-volume to re-mount the cinder volume, because that would also format it:

  • manually add entry in /etc/fstab, following the pattern used in the script above.
blkid -o export /dev/sdb | grep -E '^(UUID|TYPE)='

Add this line: UUID=a81014b1-7b19-44b6-ab4d-b22a30ddb637 /srv ext4 rw,discard,rw,relatime 0 2 to /etc/fstab to ensure it survives the next reboot.

there is a follow-up issue here: T429967