Page MenuHomePhabricator
Create Task
Maniphest T324974

Migrate SSH bastions to Bullseye
Closed, ResolvedPublic
Actions

Description

Migrations of the bastions to Bullseye:

  • bast1003 (reimaged)
  • bast2002 (reimaged)
  • bast3005 (replaced by new bast3006 VM)
  • bast4003 (replaced by new bast4004 VM)
  • bast5002 (replaced by new bast5003 VM)
  • bast6001 (replaced by new bast6002 VM)

Details

SubjectRepoBranchLines +/-
Remove bast3005/bast5002 from Puppetoperations/puppetproduction+0 -24
Remove bast4003operations/puppetproduction+0 -11
Remove previos bastions from bastion_host listoperations/puppetproduction+0 -8
Disable old bastionsoperations/puppetproduction+4 -4
Add new bastionsoperations/puppetproduction+20 -12
Add new bastions in esams/eqsin/drmrsoperations/puppetproduction+3 -3
Add bast5003operations/puppetproduction+11 -1
Add bast3006/bast4004/bast6002 to Puppetoperations/puppetproduction+31 -1
Customize query in gerrit

Event Timeline

MoritzMuehlenhoff created this task.Dec 12 2022, 1:26 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 12 2022, 1:26 PM
MoritzMuehlenhoff claimed this task.Dec 12 2022, 1:27 PM
MoritzMuehlenhoff triaged this task as Medium priority.
gerritbot added a comment.Jan 11 2023, 1:55 PM

Change 878945 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add bast3006/bast4004/bast6002 to Puppet

https://gerrit.wikimedia.org/r/878945

gerritbot added a project: Patch-For-Review.Jan 11 2023, 1:55 PM
gerritbot added a comment.Jan 11 2023, 2:08 PM

Change 878945 merged by Muehlenhoff:

[operations/puppet@production] Add bast3006/bast4004/bast6002 to Puppet

https://gerrit.wikimedia.org/r/878945

Maintenance_bot removed a project: Patch-For-Review.Jan 11 2023, 2:30 PM
gerritbot added a comment.Jan 12 2023, 8:42 AM

Change 879273 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add bast5003

https://gerrit.wikimedia.org/r/879273

gerritbot added a project: Patch-For-Review.Jan 12 2023, 8:42 AM
gerritbot added a comment.Jan 12 2023, 8:46 AM

Change 879273 merged by Muehlenhoff:

[operations/puppet@production] Add bast5003

https://gerrit.wikimedia.org/r/879273

Maintenance_bot removed a project: Patch-For-Review.Jan 12 2023, 9:30 AM
Stashbot added a comment.Jan 13 2023, 9:06 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-13T09:06:41Z] <moritzm> installing bast3006 T324974

Stashbot added a comment.Jan 13 2023, 9:41 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-13T09:41:08Z] <moritzm> installing bast4004 T324974

Stashbot added a comment.Jan 13 2023, 10:53 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-13T10:53:53Z] <moritzm> installing bast5003 T324974

Stashbot added a comment.Jan 13 2023, 12:48 PM

Mentioned in SAL (#wikimedia-operations) [2023-01-13T12:48:47Z] <moritzm> installing bast6002 T324974

gerritbot added a comment.Jan 16 2023, 10:32 AM

Change 880433 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add new bastions

https://gerrit.wikimedia.org/r/880433

gerritbot added a project: Patch-For-Review.Jan 16 2023, 10:32 AM
gerritbot added a comment.Jan 16 2023, 2:09 PM

Change 880477 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add new bastions in esams/eqsin/drmrs

https://gerrit.wikimedia.org/r/880477

gerritbot added a comment.Jan 16 2023, 2:12 PM

Change 880477 merged by Muehlenhoff:

[operations/puppet@production] Add new bastions in esams/eqsin/drmrs

https://gerrit.wikimedia.org/r/880477

gerritbot added a comment.Jan 17 2023, 9:48 AM

Change 880433 merged by Muehlenhoff:

[operations/puppet@production] Add new bastions

https://gerrit.wikimedia.org/r/880433

kostajh subscribed.Jan 18 2023, 12:41 PM

I updated https://wikitech.wikimedia.org/wiki/Template:BastionMap and created pages for the new bastions, but as discussed on ops@lists.wikimedia.org, the ssh fingerprints are missing.

MoritzMuehlenhoff added a comment.Jan 18 2023, 1:45 PM
In T324974#8534698, @kostajh wrote:

I updated https://wikitech.wikimedia.org/wiki/Template:BastionMap and created pages for the new bastions, but as discussed on ops@lists.wikimedia.org, the ssh fingerprints are missing.

Thanks for that! I've backfilled the fingerprint wikitech entries now.

gerritbot added a comment.Jan 27 2023, 9:10 AM

Change 884225 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Disable old bastions

https://gerrit.wikimedia.org/r/884225

gerritbot added a comment.Jan 27 2023, 9:40 AM

Change 884225 merged by Muehlenhoff:

[operations/puppet@production] Disable old bastions

https://gerrit.wikimedia.org/r/884225

Maintenance_bot removed a project: Patch-For-Review.Jan 27 2023, 10:31 AM
gerritbot added a comment.Jan 30 2023, 10:19 AM

Change 884832 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove previos bastions from bastion_host list

https://gerrit.wikimedia.org/r/884832

gerritbot added a project: Patch-For-Review.Jan 30 2023, 10:19 AM

Change 884832 merged by Muehlenhoff:

[operations/puppet@production] Remove previos bastions from bastion_host list

https://gerrit.wikimedia.org/r/884832

Maintenance_bot removed a project: Patch-For-Review.Jan 30 2023, 10:31 AM
ops-monitoring-bot added a comment.Jan 30 2023, 10:46 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast4003.wikimedia.org

  • bast4003.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ulsfo to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ulsfo to Netbox
gerritbot added a comment.Jan 30 2023, 11:04 AM

Change 884845 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove bast4003

https://gerrit.wikimedia.org/r/884845

gerritbot added a project: Patch-For-Review.Jan 30 2023, 11:04 AM
gerritbot added a comment.Jan 30 2023, 11:12 AM

Change 884845 merged by Muehlenhoff:

[operations/puppet@production] Remove bast4003

https://gerrit.wikimedia.org/r/884845

Maintenance_bot removed a project: Patch-For-Review.Jan 30 2023, 11:30 AM
ops-monitoring-bot added a comment.Jan 30 2023, 11:57 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast6001.wikimedia.org

  • bast6001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster drmrs01 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster drmrs01 to Netbox
gerritbot added a comment.Feb 1 2023, 1:44 PM

Change 885810 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove bast3005/bast5002 from Puppet

https://gerrit.wikimedia.org/r/885810

gerritbot added a project: Patch-For-Review.Feb 1 2023, 1:44 PM
gerritbot added a comment.Feb 1 2023, 1:47 PM

Change 885810 merged by Muehlenhoff:

[operations/puppet@production] Remove bast3005/bast5002 from Puppet

https://gerrit.wikimedia.org/r/885810

ops-monitoring-bot added a comment.Feb 1 2023, 1:48 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast5002.wikimedia.org

  • bast5002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqsin to Netbox
MoritzMuehlenhoff updated the task description. (Show Details)Feb 1 2023, 1:50 PM
ops-monitoring-bot added a comment.Feb 1 2023, 2:07 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast3005.wikimedia.org

  • bast3005.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
Maintenance_bot removed a project: Patch-For-Review.Feb 1 2023, 2:30 PM
ops-monitoring-bot added a comment.Feb 7 2023, 9:02 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 9:42 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast2002.wikimedia.org with OS bullseye completed:

  • bast2002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302070902_jmm_408503_bast2002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 7 2023, 9:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by jmm@cumin2002 for host bast1003.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Feb 7 2023, 10:17 AM

Cookbook cookbooks.sre.hosts.reimage started by jmm@cumin2002 for host bast1003.wikimedia.org with OS bullseye completed:

  • bast1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302070944_jmm_416713_bast1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MoritzMuehlenhoff closed this task as Resolved.Feb 7 2023, 10:26 AM
MoritzMuehlenhoff updated the task description. (Show Details)

This is complete

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL