Page MenuHomePhabricator
Create Task
Maniphest T327867

Migrate the install servers to Bullseye
Open, MediumPublic
Actions

Description

These are still on Buster currently. Replace them with new VMs:

sitenew VM namein service
eqiadinstall1004
codfwinstall2004yes
esamsinstall3002yes
ulsfoinstall4002yes
eqsininstall5002yes
drmrsinstall6002

Details

ProjectBranchLines +/-Subject
operations/homer/publicmaster+1 -1Point DHCP server in eqiad to install1004
operations/dnsmaster+1 -1Move webproxy in eqiad to install1004
operations/puppetproduction+2 -2Remove installserver role from install5001
operations/puppetproduction+2 -2Point DHCP server for eqsin to install5002
operations/homer/publicmaster+1 -1Point DHCP server in eqsin to install5002
operations/dnsmaster+1 -1Point proxy in eqsin to install5002
operations/puppetproduction+2 -2Assign installserver role to install5002
operations/puppetproduction+2 -2Remove installserver role from install4001
operations/puppetproduction+2 -2Update DHCP config in ulsfo
operations/homer/publicmaster+1 -1Point DHCP server in ulsfo to install4002
operations/dnsmaster+1 -1Move webproxy in ulsfo to install4002
operations/puppetproduction+2 -2Setup install4002 as install server
operations/puppetproduction+6 -2Remove installserver role from install3001
operations/puppetproduction+3 -3Update DHCP config for esams
operations/homer/publicmaster+1 -1Point DHCP server in esams to install3002
operations/dnsmaster+1 -1Point the webproxy in esams to install3002
operations/puppetproduction+5 -1Apply installserver role to install3002
operations/puppetproduction+5 -1Remove installserver role from install2003
operations/homer/publicmaster+1 -1Point to install2004 for DHCP in codfw
operations/puppetproduction+16 -16Move next-server settings from install2003->2004
operations/puppetproduction+6 -6Update cloudbastion rules for install2004
operations/dnsmaster+1 -1Move webproxy.codfw.wmnet to install2004
operations/puppetproduction+5 -1Apply role::installserver to install2004
labs/privatemaster+0 -0New stub keytabs for the new install servers
operations/puppetproduction+4 -17Remove service toggle for TFTP
operations/puppetproduction+30 -0Add new install servers
Show related patches Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Wed, Jan 25, 8:59 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptWed, Jan 25, 8:59 AM
Peachey88 renamed this task from MIgrate the install servers to Bullseye to Migrate the install servers to Bullseye.Wed, Jan 25, 8:59 AM
gerritbot added a comment.Wed, Jan 25, 2:52 PM

Change 883581 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add new install servers

https://gerrit.wikimedia.org/r/883581

gerritbot added a project: Patch-For-Review.Wed, Jan 25, 2:52 PM
gerritbot added a comment.Thu, Jan 26, 8:07 AM

Change 883581 merged by Muehlenhoff:

[operations/puppet@production] Add new install servers

https://gerrit.wikimedia.org/r/883581

Maintenance_bot removed a project: Patch-For-Review.Thu, Jan 26, 8:30 AM
Stashbot added a comment.Fri, Jan 27, 1:08 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-27T13:08:26Z] <moritzm> installing install6002 T327867

Stashbot added a comment.Fri, Jan 27, 1:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-27T13:46:35Z] <moritzm> installing install5002 T327867

gerritbot added a comment.Fri, Jan 27, 2:15 PM

Change 884310 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove service toggle for TFTP

https://gerrit.wikimedia.org/r/884310

gerritbot added a project: Patch-For-Review.Fri, Jan 27, 2:15 PM
Stashbot added a comment.Fri, Jan 27, 2:40 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-27T14:40:25Z] <moritzm> installing install3002 T327867

Stashbot added a comment.Mon, Jan 30, 7:50 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-30T07:50:15Z] <moritzm> installing install2004 T327867

Stashbot added a comment.Mon, Jan 30, 8:48 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-30T08:48:46Z] <moritzm> installing install1004 T327867

Stashbot added a comment.Mon, Jan 30, 11:42 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-30T11:42:57Z] <moritzm> installing install4002 T327867

gerritbot added a comment.Mon, Jan 30, 7:19 PM

Change 884310 merged by Dzahn:

[operations/puppet@production] Remove service toggle for TFTP

https://gerrit.wikimedia.org/r/884310

Maintenance_bot removed a project: Patch-For-Review.Mon, Jan 30, 7:30 PM
gerritbot added a comment.Tue, Jan 31, 6:32 AM

Change 885246 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Apply role::installserver to install2004

https://gerrit.wikimedia.org/r/885246

gerritbot added a project: Patch-For-Review.Tue, Jan 31, 6:32 AM
gerritbot added a comment.Tue, Jan 31, 7:00 AM

Change 885263 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] New stub keytabs for the new install servers

https://gerrit.wikimedia.org/r/885263

gerritbot added a comment.Tue, Jan 31, 7:22 AM

Change 885263 merged by Muehlenhoff:

[labs/private@master] New stub keytabs for the new install servers

https://gerrit.wikimedia.org/r/885263

Muehlenhoff mentioned this in rLPRI1dbad0e297c8: New stub keytabs for the new install servers.Tue, Jan 31, 7:24 AM
gerritbot added a comment.Tue, Jan 31, 9:29 AM

Change 885246 merged by Muehlenhoff:

[operations/puppet@production] Apply role::installserver to install2004

https://gerrit.wikimedia.org/r/885246

Maintenance_bot removed a project: Patch-For-Review.Tue, Jan 31, 9:30 AM
gerritbot added a comment.Tue, Jan 31, 10:03 AM

Change 885285 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Move webproxy.codfw.wmnet to install2004

https://gerrit.wikimedia.org/r/885285

gerritbot added a project: Patch-For-Review.Tue, Jan 31, 10:03 AM
gerritbot added a comment.Tue, Jan 31, 11:06 AM

Change 885285 merged by Muehlenhoff:

[operations/dns@master] Move webproxy.codfw.wmnet to install2004

https://gerrit.wikimedia.org/r/885285

Maintenance_bot removed a project: Patch-For-Review.Tue, Jan 31, 11:29 AM
gerritbot added a comment.Tue, Jan 31, 12:11 PM

Change 885304 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update cloudbastion rules for install2004

https://gerrit.wikimedia.org/r/885304

gerritbot added a project: Patch-For-Review.Tue, Jan 31, 12:11 PM
gerritbot added a comment.Tue, Jan 31, 12:15 PM

Change 885326 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point to install2004 for DHCP in codfw

https://gerrit.wikimedia.org/r/885326

gerritbot added a comment.Tue, Jan 31, 12:54 PM

Change 885336 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Move next-server settings from install2003->2004

https://gerrit.wikimedia.org/r/885336

MoritzMuehlenhoff added a comment.Tue, Jan 31, 12:57 PM

install2004 has had the installserver role assigned and it's now acting the web proxy for codfw. The DHCP server is currrently stopped, tomorrow morning I'll merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/885336 and https://gerrit.wikimedia.org/r/c/operations/homer/public/+/885326 and test a reimage and VM installation.

gerritbot added a comment.Tue, Jan 31, 3:37 PM

Change 885304 merged by Muehlenhoff:

[operations/puppet@production] Update cloudbastion rules for install2004

https://gerrit.wikimedia.org/r/885304

gerritbot added a comment.Wed, Feb 1, 8:44 AM

Change 885336 merged by Muehlenhoff:

[operations/puppet@production] Move next-server settings from install2003->2004

https://gerrit.wikimedia.org/r/885336

MoritzMuehlenhoff updated the task description. (Show Details)Wed, Feb 1, 8:46 AM
gerritbot added a comment.Wed, Feb 1, 8:54 AM

Change 885326 merged by Muehlenhoff:

[operations/homer/public@master] Point to install2004 for DHCP in codfw

https://gerrit.wikimedia.org/r/885326

Maintenance_bot removed a project: Patch-For-Review.Wed, Feb 1, 9:30 AM
ops-monitoring-bot added a comment.Wed, Feb 1, 11:27 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm2001.codfw.wmnet

  • testvm2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
ops-monitoring-bot added a comment.Wed, Feb 1, 12:41 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm2003.codfw.wmnet

  • testvm2003.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
gerritbot added a comment.Wed, Feb 1, 12:48 PM

Change 885799 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install2003

https://gerrit.wikimedia.org/r/885799

gerritbot added a project: Patch-For-Review.Wed, Feb 1, 12:48 PM
gerritbot added a comment.Wed, Feb 1, 12:51 PM

Change 885799 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install2003

https://gerrit.wikimedia.org/r/885799

MoritzMuehlenhoff added a comment.Wed, Feb 1, 12:55 PM

install2004 is fully in service now, I have tested a reimage on baremetal and VM installation successfully. atftpd, dhcpd and nginx have been stopped on install2003, but I'm keeping the VM for a few more days just in case.

MoritzMuehlenhoff triaged this task as Medium priority.Wed, Feb 1, 12:56 PM
MoritzMuehlenhoff updated the task description. (Show Details)
gerritbot added a comment.Wed, Feb 1, 1:18 PM

Change 885805 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in esams to install3002

https://gerrit.wikimedia.org/r/885805

gerritbot added a comment.Wed, Feb 1, 1:18 PM

Change 885806 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in ulsfo to install4002

https://gerrit.wikimedia.org/r/885806

gerritbot added a comment.Thu, Feb 2, 7:55 AM

Change 885982 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point the webproxy in esams to install3002

https://gerrit.wikimedia.org/r/885982

gerritbot added a comment.Thu, Feb 2, 7:56 AM

Change 885983 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Apply installserver role to install3002

https://gerrit.wikimedia.org/r/885983

gerritbot added a comment.Thu, Feb 2, 8:00 AM

Change 885983 merged by Muehlenhoff:

[operations/puppet@production] Apply installserver role to install3002

https://gerrit.wikimedia.org/r/885983

gerritbot added a comment.Thu, Feb 2, 8:17 AM

Change 885984 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update DHCP config for esams

https://gerrit.wikimedia.org/r/885984

gerritbot added a comment.Thu, Feb 2, 8:43 AM

Change 885982 merged by Muehlenhoff:

[operations/dns@master] Point the webproxy in esams to install3002

https://gerrit.wikimedia.org/r/885982

gerritbot added a comment.Thu, Feb 2, 8:44 AM

Change 885805 merged by jenkins-bot:

[operations/homer/public@master] Point DHCP server in esams to install3002

https://gerrit.wikimedia.org/r/885805

gerritbot added a comment.Thu, Feb 2, 8:57 AM

Change 885984 merged by Muehlenhoff:

[operations/puppet@production] Update DHCP config for esams

https://gerrit.wikimedia.org/r/885984

gerritbot added a comment.Thu, Feb 2, 9:04 AM

Change 885989 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install3001

https://gerrit.wikimedia.org/r/885989

gerritbot added a comment.Thu, Feb 2, 9:53 AM

Change 885989 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install3001

https://gerrit.wikimedia.org/r/885989

MoritzMuehlenhoff updated the task description. (Show Details)Thu, Feb 2, 10:02 AM
gerritbot added a comment.Thu, Feb 2, 12:41 PM

Change 886031 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Move webproxy in ulsfo to install4002

https://gerrit.wikimedia.org/r/886031

gerritbot added a comment.Thu, Feb 2, 12:53 PM

Change 886036 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Setup install4002 as install server

https://gerrit.wikimedia.org/r/886036

gerritbot added a comment.Thu, Feb 2, 12:54 PM

Change 886037 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update DHCP config in ulsfo

https://gerrit.wikimedia.org/r/886037

gerritbot added a comment.Thu, Feb 2, 1:11 PM

Change 886036 merged by Muehlenhoff:

[operations/puppet@production] Setup install4002 as install server

https://gerrit.wikimedia.org/r/886036

gerritbot added a comment.Thu, Feb 2, 1:38 PM

Change 886031 merged by Muehlenhoff:

[operations/dns@master] Move webproxy in ulsfo to install4002

https://gerrit.wikimedia.org/r/886031

gerritbot added a comment.Thu, Feb 2, 1:42 PM

Change 885806 merged by Muehlenhoff:

[operations/homer/public@master] Point DHCP server in ulsfo to install4002

https://gerrit.wikimedia.org/r/885806

gerritbot added a comment.Thu, Feb 2, 1:55 PM

Change 886037 merged by Muehlenhoff:

[operations/puppet@production] Update DHCP config in ulsfo

https://gerrit.wikimedia.org/r/886037

gerritbot added a comment.Thu, Feb 2, 2:02 PM

Change 886049 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install4001

https://gerrit.wikimedia.org/r/886049

gerritbot added a comment.Thu, Feb 2, 2:06 PM

Change 886049 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install4001

https://gerrit.wikimedia.org/r/886049

MoritzMuehlenhoff updated the task description. (Show Details)Thu, Feb 2, 2:12 PM
gerritbot added a comment.Thu, Feb 2, 2:15 PM

Change 886053 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in eqsin to install5002

https://gerrit.wikimedia.org/r/886053

ayounsi mentioned this in T327812: eqsin hosts are not rebooting when running sre.hosts.reimage cookbook.Fri, Feb 3, 7:45 AM
gerritbot added a comment.Fri, Feb 3, 8:16 AM

Change 886313 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point proxy in eqsin to install5002

https://gerrit.wikimedia.org/r/886313

gerritbot added a comment.Fri, Feb 3, 8:17 AM

Change 886314 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Assign installserver role to install5002

https://gerrit.wikimedia.org/r/886314

gerritbot added a comment.Fri, Feb 3, 8:24 AM

Change 886314 merged by Muehlenhoff:

[operations/puppet@production] Assign installserver role to install5002

https://gerrit.wikimedia.org/r/886314

gerritbot added a comment.Fri, Feb 3, 9:31 AM

Change 886313 merged by Muehlenhoff:

[operations/dns@master] Point proxy in eqsin to install5002

https://gerrit.wikimedia.org/r/886313

gerritbot added a comment.Fri, Feb 3, 9:36 AM

Change 886320 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Point DHCP server for eqsin to install5002

https://gerrit.wikimedia.org/r/886320

gerritbot added a comment.Fri, Feb 3, 9:41 AM

Change 886053 merged by Muehlenhoff:

[operations/homer/public@master] Point DHCP server in eqsin to install5002

https://gerrit.wikimedia.org/r/886053

gerritbot added a comment.Fri, Feb 3, 9:52 AM

Change 886320 merged by Muehlenhoff:

[operations/puppet@production] Point DHCP server for eqsin to install5002

https://gerrit.wikimedia.org/r/886320

gerritbot added a comment.Fri, Feb 3, 10:10 AM

Change 886326 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove installserver role from install5001

https://gerrit.wikimedia.org/r/886326

gerritbot added a comment.Fri, Feb 3, 10:13 AM

Change 886326 merged by Muehlenhoff:

[operations/puppet@production] Remove installserver role from install5001

https://gerrit.wikimedia.org/r/886326

MoritzMuehlenhoff updated the task description. (Show Details)Fri, Feb 3, 10:20 AM
Maintenance_bot removed a project: Patch-For-Review.Fri, Feb 3, 10:30 AM
gerritbot added a comment.Mon, Feb 6, 12:57 PM

Change 886888 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/homer/public@master] Point DHCP server in eqiad to install1004

https://gerrit.wikimedia.org/r/886888

gerritbot added a project: Patch-For-Review.Mon, Feb 6, 12:57 PM

Change 886889 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Move webproxy in eqiad to install1004

https://gerrit.wikimedia.org/r/886889

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL