Page MenuHomePhabricator

Migrate the install servers to Bullseye
Closed, ResolvedPublic

Description

These are still on Buster currently. Replace them with new VMs:

sitenew VM namein serviceold VM decommissioned
eqiadinstall1004yesyes
codfwinstall2004yesyes
esamsinstall3002yesyes
ulsfoinstall4002yesyes
eqsininstall5002yesyes
drmrsinstall6002yesyes

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+0 -18
labs/privatemaster+0 -0
operations/puppetproduction+0 -32
operations/puppetproduction+1 -5
operations/puppetproduction+2 -2
operations/puppetproduction+42 -42
operations/homer/publicmaster+1 -1
operations/dnsmaster+1 -1
operations/puppetproduction+2 -6
operations/puppetproduction+1 -9
operations/puppetproduction+9 -0
operations/puppetproduction+4 -4
operations/dnsmaster+1 -1
operations/puppetproduction+1 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/homer/publicmaster+1 -1
operations/dnsmaster+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/puppetproduction+2 -2
operations/homer/publicmaster+1 -1
operations/dnsmaster+1 -1
operations/puppetproduction+2 -2
operations/puppetproduction+6 -2
operations/puppetproduction+3 -3
operations/homer/publicmaster+1 -1
operations/dnsmaster+1 -1
operations/puppetproduction+5 -1
operations/puppetproduction+5 -1
operations/homer/publicmaster+1 -1
operations/puppetproduction+16 -16
operations/puppetproduction+6 -6
operations/dnsmaster+1 -1
operations/puppetproduction+5 -1
labs/privatemaster+0 -0
operations/puppetproduction+4 -17
operations/puppetproduction+30 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 885336 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move next-server settings from install2003->2004

https://gerrit.wikimedia.org/r/885336

install2004 has had the installserver role assigned and it's now acting the web proxy for codfw. The DHCP server is currrently stopped, tomorrow morning I'll merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/885336 and https://gerrit.wikimedia.org/r/c/operations/homer/public/+/885326 and test a reimage and VM installation.

Change 885304 merged by Muehlenhoff:

[operations/puppet@production] Update cloudbastion rules for install2004

https://gerrit.wikimedia.org/r/885304

Change 885336 merged by Muehlenhoff:

[operations/puppet@production] Move next-server settings from install2003->2004

https://gerrit.wikimedia.org/r/885336

Change 885326 merged by Muehlenhoff:

[operations/homer/public@master] Point to install2004 for DHCP in codfw

https://gerrit.wikimedia.org/r/885326

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm2001.codfw.wmnet

  • testvm2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm2003.codfw.wmnet

  • testvm2003.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox

Change 885799 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install2003

https://gerrit.wikimedia.org/r/885799

Change 885799 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install2003

https://gerrit.wikimedia.org/r/885799

install2004 is fully in service now, I have tested a reimage on baremetal and VM installation successfully. atftpd, dhcpd and nginx have been stopped on install2003, but I'm keeping the VM for a few more days just in case.

MoritzMuehlenhoff updated the task description. (Show Details)

Change 885805 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in esams to install3002

https://gerrit.wikimedia.org/r/885805

Change 885806 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in ulsfo to install4002

https://gerrit.wikimedia.org/r/885806

Change 885982 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point the webproxy in esams to install3002

https://gerrit.wikimedia.org/r/885982

Change 885983 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Apply installserver role to install3002

https://gerrit.wikimedia.org/r/885983

Change 885983 merged by Muehlenhoff:

[operations/puppet@production] Apply installserver role to install3002

https://gerrit.wikimedia.org/r/885983

Change 885984 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update DHCP config for esams

https://gerrit.wikimedia.org/r/885984

Change 885982 merged by Muehlenhoff:

[operations/dns@master] Point the webproxy in esams to install3002

https://gerrit.wikimedia.org/r/885982

Change 885805 merged by jenkins-bot:

[operations/homer/public@master] Point DHCP server in esams to install3002

https://gerrit.wikimedia.org/r/885805

Change 885984 merged by Muehlenhoff:

[operations/puppet@production] Update DHCP config for esams

https://gerrit.wikimedia.org/r/885984

Change 885989 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install3001

https://gerrit.wikimedia.org/r/885989

Change 885989 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install3001

https://gerrit.wikimedia.org/r/885989

Change 886031 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Move webproxy in ulsfo to install4002

https://gerrit.wikimedia.org/r/886031

Change 886036 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Setup install4002 as install server

https://gerrit.wikimedia.org/r/886036

Change 886037 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update DHCP config in ulsfo

https://gerrit.wikimedia.org/r/886037

Change 886036 merged by Muehlenhoff:

[operations/puppet@production] Setup install4002 as install server

https://gerrit.wikimedia.org/r/886036

Change 886031 merged by Muehlenhoff:

[operations/dns@master] Move webproxy in ulsfo to install4002

https://gerrit.wikimedia.org/r/886031

Change 885806 merged by Muehlenhoff:

[operations/homer/public@master] Point DHCP server in ulsfo to install4002

https://gerrit.wikimedia.org/r/885806

Change 886037 merged by Muehlenhoff:

[operations/puppet@production] Update DHCP config in ulsfo

https://gerrit.wikimedia.org/r/886037

Change 886049 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install4001

https://gerrit.wikimedia.org/r/886049

Change 886049 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install4001

https://gerrit.wikimedia.org/r/886049

Change 886053 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in eqsin to install5002

https://gerrit.wikimedia.org/r/886053

Change 886313 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point proxy in eqsin to install5002

https://gerrit.wikimedia.org/r/886313

Change 886314 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign installserver role to install5002

https://gerrit.wikimedia.org/r/886314

Change 886314 merged by Muehlenhoff:

[operations/puppet@production] Assign installserver role to install5002

https://gerrit.wikimedia.org/r/886314

Change 886313 merged by Muehlenhoff:

[operations/dns@master] Point proxy in eqsin to install5002

https://gerrit.wikimedia.org/r/886313

Change 886320 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Point DHCP server for eqsin to install5002

https://gerrit.wikimedia.org/r/886320

Change 886053 merged by Muehlenhoff:

[operations/homer/public@master] Point DHCP server in eqsin to install5002

https://gerrit.wikimedia.org/r/886053

Change 886320 merged by Muehlenhoff:

[operations/puppet@production] Point DHCP server for eqsin to install5002

https://gerrit.wikimedia.org/r/886320

Change 886326 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install5001

https://gerrit.wikimedia.org/r/886326

Change 886326 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install5001

https://gerrit.wikimedia.org/r/886326

Change 886888 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in eqiad to install1004

https://gerrit.wikimedia.org/r/886888

Change 886889 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Move webproxy in eqiad to install1004

https://gerrit.wikimedia.org/r/886889

Change 887302 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign installserver role to install6002

https://gerrit.wikimedia.org/r/887302

Change 887302 merged by Muehlenhoff:

[operations/puppet@production] Assign installserver role to install6002

https://gerrit.wikimedia.org/r/887302

Change 887304 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point webproxy in drmrs to install6002

https://gerrit.wikimedia.org/r/887304

Change 887305 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update tftp server settings for drmrs

https://gerrit.wikimedia.org/r/887305

Change 887304 merged by Muehlenhoff:

[operations/dns@master] Point webproxy in drmrs to install6002

https://gerrit.wikimedia.org/r/887304

Change 887305 merged by Muehlenhoff:

[operations/puppet@production] Update tftp server settings for drmrs

https://gerrit.wikimedia.org/r/887305

Change 887310 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add testvm6001

https://gerrit.wikimedia.org/r/887310

Change 887310 merged by Muehlenhoff:

[operations/puppet@production] Add testvm6001

https://gerrit.wikimedia.org/r/887310

Change 887311 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Drop installserver role from install6001

https://gerrit.wikimedia.org/r/887311

Change 887311 merged by Muehlenhoff:

[operations/puppet@production] Drop installserver role from install6001

https://gerrit.wikimedia.org/r/887311

Mentioned in SAL (#wikimedia-operations) [2023-02-08T14:29:37Z] <moritzm> test install of testvm6001 T327867

Change 887790 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign installserver role to install1004

https://gerrit.wikimedia.org/r/887790

Change 887790 merged by Muehlenhoff:

[operations/puppet@production] Assign installserver role to install1004

https://gerrit.wikimedia.org/r/887790

Change 886889 merged by Muehlenhoff:

[operations/dns@master] Move webproxy in eqiad to install1004

https://gerrit.wikimedia.org/r/886889

Change 886888 merged by Muehlenhoff:

[operations/homer/public@master] Point DHCP server in eqiad to install1004

https://gerrit.wikimedia.org/r/886888

Change 887796 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update next-server DHCP settings towards install1004

https://gerrit.wikimedia.org/r/887796

Change 887796 merged by Muehlenhoff:

[operations/puppet@production] Update next-server DHCP settings towards install1004

https://gerrit.wikimedia.org/r/887796

Change 887798 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update cloud proxies

https://gerrit.wikimedia.org/r/887798

Change 887798 merged by Muehlenhoff:

[operations/puppet@production] Update cloud proxies

https://gerrit.wikimedia.org/r/887798

Change 887941 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install1003

https://gerrit.wikimedia.org/r/887941

Change 887941 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install1003

https://gerrit.wikimedia.org/r/887941

All install servers are running Bullseye now, the only missing bit is to remove the old VMs.

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install6001.wikimedia.org

  • install6001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster drmrs01 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster drmrs01 to Netbox

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install5001.wikimedia.org

  • install5001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install4001.wikimedia.org

  • install4001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ulsfo to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ulsfo to Netbox

Change 888192 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove Puppet references for install[3456]001

https://gerrit.wikimedia.org/r/888192

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install3001.wikimedia.org

  • install3001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox

Change 888192 merged by Muehlenhoff:

[operations/puppet@production] Remove Puppet references for install[3456]001

https://gerrit.wikimedia.org/r/888192

Change 888196 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Remove obsolete stub keytabs

https://gerrit.wikimedia.org/r/888196

Change 888196 merged by Muehlenhoff:

[labs/private@master] Remove obsolete stub keytabs

https://gerrit.wikimedia.org/r/888196

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install2003.wikimedia.org

  • install2003.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox

Change 888656 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove Puppet references to install[12]003

https://gerrit.wikimedia.org/r/888656

Change 888656 merged by Muehlenhoff:

[operations/puppet@production] Remove Puppet references to install[12]003

https://gerrit.wikimedia.org/r/888656

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install1003.wikimedia.org

  • install1003.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
MoritzMuehlenhoff claimed this task.
MoritzMuehlenhoff updated the task description. (Show Details)

All install servers are now on Bullseye.

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm6001.drmrs.wmnet

  • testvm6001.drmrs.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster drmrs01 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster drmrs01 to Netbox