Page MenuHomePhabricator

Upgrade docker-registry servers to Debian Buster
Closed, ResolvedPublic

Description

Hosts: registry[2001-2002].codfw.wmnet,registry[1001-1002].eqiad.wmnet

These are currently Debian Stretch, we should upgrade them to Buster. I ran into this when working on T179696, especially because the docker-report library being used is intended for Buster only.

Event Timeline

Skimming the puppet role, there's:

# this could be removed when buster or next debian includes a 2.7+ version
apt::pin { 'strech_wikimedia_docker_registry_27':
    package  => 'docker-registry',
    pin      => 'version 2.7.0~rc0~wmf1-1',
    priority => '1002',
}

buster has 2.6.2~ds1-2, so it sounds like we might need to backport 2.7.1+ds2-7 from bullseye. Or would we rather carry the stretch packages forward to buster, since it's all golang anyways?

Mentioned in SAL (#wikimedia-operations) [2021-03-02T21:51:00Z] <legoktm> copied docker-registry package from stretch-wikimedia to buster-wikimedia (T272550)

registry1003 is now pooled, I did a test pull to it specifically and it worked fine.

Mentioned in SAL (#wikimedia-operations) [2021-03-05T00:59:46Z] <legoktm> depooled registry1001/registry1002 (old stretch VMs) - T272550

Update: 2 new buster VMs in eqiad are running, and I depooled the 2 stretch ones, will delete them on Monday if no other problems arise.

In codfw 1 buster VM is running alongside the 2 stretch ones, except I accidentally created the second new VM with the wrong name (registry2004.eqiad.wmnet) ... waiting for some guidance on how to delete that before creating a new VM.

In codfw 1 buster VM is running alongside the 2 stretch ones, except I accidentally created the second new VM with the wrong name (registry2004.eqiad.wmnet) ... waiting for some guidance on how to delete that before creating a new VM.

I would assume you can just use https://wikitech.wikimedia.org/wiki/Decom_script to get rid of it.

In codfw 1 buster VM is running alongside the 2 stretch ones, except I accidentally created the second new VM with the wrong name (registry2004.eqiad.wmnet) ... waiting for some guidance on how to delete that before creating a new VM.

I would assume you can just use https://wikitech.wikimedia.org/wiki/Decom_script to get rid of it.

Had to fix the script with https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/668572 but now it's gone :)


codfw has 2 buster VMs running now, I depooled the 2 stretch ones.

Change 670604 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] Remove registry1001.eqiad.wmnet

https://gerrit.wikimedia.org/r/670604

cookbooks.sre.hosts.decommission executed by legoktm@cumin1001 for hosts: registry1001.eqiad.wmnet

  • registry1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox

Change 670604 merged by Legoktm:
[operations/puppet@production] Remove registry1001.eqiad.wmnet

https://gerrit.wikimedia.org/r/670604

Change 670605 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] Remove registry200[12].codfw.wmnet

https://gerrit.wikimedia.org/r/670605

cookbooks.sre.hosts.decommission executed by legoktm@cumin1001 for hosts: registry[2001-2002].codfw.wmnet

  • registry2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • registry2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox

Change 670605 merged by Legoktm:
[operations/puppet@production] Remove registry200[12].codfw.wmnet

https://gerrit.wikimedia.org/r/670605

Change 670607 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] Remove registry1002.eqiad.wmnet

https://gerrit.wikimedia.org/r/670607

Change 670608 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[operations/puppet@production] site.pp: Tighten registry* regex

https://gerrit.wikimedia.org/r/670608

cookbooks.sre.hosts.decommission executed by legoktm@cumin1001 for hosts: registry1002.eqiad.wmnet

  • registry1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox

Change 670607 merged by Legoktm:
[operations/puppet@production] Remove registry1002.eqiad.wmnet

https://gerrit.wikimedia.org/r/670607

Change 670608 merged by Legoktm:
[operations/puppet@production] site.pp: Tighten registry* regex

https://gerrit.wikimedia.org/r/670608

Everything is Buster now, Stretch is gone \o/