Page MenuHomePhabricator
Create Task
Maniphest T304598

cloudgw: upgrade servers to Debian 11 Bullseye
Closed, ResolvedPublic
Actions

Description

They are currently running Debian 10 Buster.

  • cloudgw2001-dev.codfw.wmnet
  • cloudgw2002-dev.codfw.wmnet
  • cloudgw1001.eqiad.wmnet
  • cloudgw1002.eqiad.wmnet

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
cloudgw1002: rename interface namesoperations/puppetproduction+10 -11
cloudgw: conntrackd: refresh NIC nameoperations/puppetproduction+1 -1
cloudgw1001: use a custom name for the dataplane NICoperations/puppetproduction+23 -5
cloudgw1001: hieradata: refresh NIC namesoperations/puppetproduction+2 -2
cloudgw: don't install kernel or nft from backportsoperations/puppetproduction+28 -21
keepalived: use version from bullseye-bpooperations/puppetproduction+19 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

• aborrero created this task.Mar 24 2022, 12:51 PM
ops-monitoring-bot added a comment.Mar 24 2022, 12:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudgw2001-dev.codfw.wmnet with OS bullseye

• aborrero changed the task status from Open to In Progress.Mar 24 2022, 12:54 PM
• aborrero triaged this task as Low priority.
• aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
ops-monitoring-bot added a comment.Mar 24 2022, 1:34 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudgw2001-dev.codfw.wmnet with OS bullseye executed with errors:

  • cloudgw2001-dev (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203241253_aborrero_2900873_cloudgw2001-dev.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Mar 24 2022, 1:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin2002 for host cloudgw2002-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2022, 2:11 PM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin2002 for host cloudgw2002-dev.codfw.wmnet with OS bullseye completed:

  • cloudgw2002-dev (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203241341_aborrero_2907817_cloudgw2002-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Mar 24 2022, 4:35 PM

Change 773585 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] keepalived: use version from bullseye-bpo

https://gerrit.wikimedia.org/r/773585

gerritbot added a project: Patch-For-Review.Mar 24 2022, 4:35 PM

Change 773586 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: don't install kernel or nft from backports

https://gerrit.wikimedia.org/r/773586

• aborrero updated the task description. (Show Details)Mar 24 2022, 4:35 PM
• aborrero added a comment.Mar 24 2022, 4:53 PM

Will wait a few more days before upgrading eqiad server, to run a few more tests, merge these patches, etc.

gerritbot added a comment.Mar 25 2022, 10:50 AM

Change 773585 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] keepalived: use version from bullseye-bpo

https://gerrit.wikimedia.org/r/773585

gerritbot added a comment.Mar 25 2022, 10:50 AM

Change 773586 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: don't install kernel or nft from backports

https://gerrit.wikimedia.org/r/773586

Maintenance_bot removed a project: Patch-For-Review.Mar 25 2022, 11:10 AM
ops-monitoring-bot added a comment.Apr 5 2022, 8:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 5 2022, 8:28 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudgw1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 5 2022, 8:35 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye

• aborrero updated the task description. (Show Details)Apr 5 2022, 9:04 AM
gerritbot added a comment.Apr 5 2022, 9:11 AM

Change 777313 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw1001: hieradata: refresh NIC names

https://gerrit.wikimedia.org/r/777313

gerritbot added a project: Patch-For-Review.Apr 5 2022, 9:11 AM

Change 777313 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw1001: hieradata: refresh NIC names

https://gerrit.wikimedia.org/r/777313

• aborrero added a comment.Apr 5 2022, 9:17 AM

The reimage resulted in new NIC names for cloudgw :-( the newer ones are longer and don't support the vlan tag attached to them.

Maintenance_bot removed a project: Patch-For-Review.Apr 5 2022, 9:31 AM
gerritbot added a comment.Apr 5 2022, 9:55 AM

Change 777333 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw1001: use a custom name for the dataplane NIC

https://gerrit.wikimedia.org/r/777333

gerritbot added a project: Patch-For-Review.Apr 5 2022, 9:55 AM
gerritbot added a comment.Apr 5 2022, 10:17 AM

Change 777333 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw1001: use a custom name for the dataplane NIC

https://gerrit.wikimedia.org/r/777333

gerritbot added a comment.Apr 5 2022, 10:22 AM

Change 777335 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw: conntrackd: refresh NIC name

https://gerrit.wikimedia.org/r/777335

gerritbot added a comment.Apr 5 2022, 10:23 AM

Change 777335 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw: conntrackd: refresh NIC name

https://gerrit.wikimedia.org/r/777335

ops-monitoring-bot added a comment.Apr 5 2022, 10:30 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudgw1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204050834_aborrero_3071366_cloudgw1001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 5 2022, 10:30 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 5 2022, 10:55 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye completed:

  • cloudgw1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204051030_aborrero_3141213_cloudgw1001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• aborrero added a comment.Apr 5 2022, 12:33 PM
aborrero@cloudgw1001:~ 4 $ sudo systemctl status systemd-sysctl
● systemd-sysctl.service - Apply Kernel Variables
     Loaded: loaded (/lib/systemd/system/systemd-sysctl.service; static)
     Active: active (exited) since Tue 2022-04-05 12:16:43 UTC; 10min ago
       Docs: man:systemd-sysctl.service(8)
             man:sysctl.d(5)
    Process: 434 ExecStart=/lib/systemd/systemd-sysctl (code=exited, status=0/SUCCESS)
   Main PID: 434 (code=exited, status=0/SUCCESS)
        CPU: 15ms

Apr 05 12:16:43 cloudgw1001 systemd[1]: Starting Apply Kernel Variables...
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '1' to 'net/ipv4/conf/dataplane.1107/forwarding', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '0' to 'net/ipv4/conf/dataplane.1107/rp_filter', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '1' to 'net/ipv4/conf/dataplane.1120/forwarding', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '0' to 'net/ipv4/conf/dataplane.1120/rp_filter', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '0' to 'net/ipv6/conf/dataplane.1107/accept_ra', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '1' to 'net/ipv6/conf/dataplane.1107/forwarding', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '0' to 'net/ipv6/conf/dataplane.1120/accept_ra', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd-sysctl[434]: Couldn't write '1' to 'net/ipv6/conf/dataplane.1120/forwarding', ignoring: No such file or directory
Apr 05 12:16:43 cloudgw1001 systemd[1]: Finished Apply Kernel Variables.

this is likely a race condition: systemd-sysctl runs before the NIC gets renamed, so the network config can't be applied

• aborrero mentioned this in T305494: interface renaming via systemd .link file can race with sysctl parameters.Apr 5 2022, 5:28 PM
ops-monitoring-bot added a comment.Apr 6 2022, 8:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 6 2022, 8:35 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1001.eqiad.wmnet with OS bullseye completed:

  • cloudgw1001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204060806_aborrero_3616294_cloudgw1001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 6 2022, 9:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 6 2022, 9:34 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye executed with errors:

  • cloudgw1002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 6 2022, 9:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 6 2022, 9:42 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye executed with errors:

  • cloudgw1002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 6 2022, 9:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye

gerritbot added a comment.Apr 6 2022, 10:21 AM

Change 777763 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] cloudgw1002: rename interface names

https://gerrit.wikimedia.org/r/777763

ops-monitoring-bot added a comment.Apr 6 2022, 10:24 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye executed with errors:

  • cloudgw1002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204060951_aborrero_3667008_cloudgw1002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Apr 6 2022, 11:20 AM

Change 777763 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] cloudgw1002: rename interface names

https://gerrit.wikimedia.org/r/777763

ops-monitoring-bot added a comment.Apr 6 2022, 11:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 6 2022, 11:48 AM

Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudgw1002.eqiad.wmnet with OS bullseye completed:

  • cloudgw1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204061123_aborrero_3754913_cloudgw1002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
• aborrero closed this task as Resolved.Apr 6 2022, 12:22 PM
• aborrero closed subtask T305494: interface renaming via systemd .link file can race with sysctl parameters as Resolved.
• aborrero updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits