Page MenuHomePhabricator
Create Task
Maniphest T303171

Upgrade s1 to Bullseye
Closed, ResolvedPublic
Actions

Description

Hosts handled elsewhere:

Special hosts:

  • db1118: old primary (T301312)
  • db1133: test host
  • db1106: sanitarium master
  • db1154: sanitarium
  • db2103: primary
  • db2072: sanitarium master
  • db2094: sanitarium
  • db2102: test host

Generated on cumin1001 with:

sudo python3 ~kormat/bin/cumin-report.py \
  "A:db-section-s1 or P{db1133.eqiad.wmnet or db2102.codfw.wmnet}" \
  "[ \$(lsb_release -sc) = 'bullseye' ] && echo 'OK' || { echo 'NOT OK'; exit 1; }"
  • clouddb1013.eqiad.wmnet
  • clouddb1017.eqiad.wmnet
  • clouddb1021.eqiad.wmnet
  • db1099.eqiad.wmnet
  • db1105.eqiad.wmnet
  • db1106.eqiad.wmnet
  • db1118.eqiad.wmnet
  • db1119.eqiad.wmnet
  • db1132.eqiad.wmnet
  • db1133.eqiad.wmnet
  • db1134.eqiad.wmnet
  • db1135.eqiad.wmnet
  • db1139.eqiad.wmnet
  • db1140.eqiad.wmnet
  • db1154.eqiad.wmnet
  • db1163.eqiad.wmnet
  • db1164.eqiad.wmnet
  • db1169.eqiad.wmnet
  • db1184.eqiad.wmnet
  • db2071.codfw.wmnet
  • db2072.codfw.wmnet
  • db2085.codfw.wmnet
  • db2088.codfw.wmnet
  • db2092.codfw.wmnet
  • db2094.codfw.wmnet
  • db2097.codfw.wmnet
  • db2102.codfw.wmnet
  • db2103.codfw.wmnet
  • db2112.codfw.wmnet
  • db2116.codfw.wmnet
  • db2130.codfw.wmnet
  • db2141.codfw.wmnet
  • db2145.codfw.wmnet
  • db2146.codfw.wmnet
  • dbstore1003.eqiad.wmnet

Details

SubjectRepoBranchLines +/-
db1106: Disable notificationoperations/puppetproduction+2 -0
db1184: Disable notificationoperations/puppetproduction+1 -0
db1135: Disable notificationoperations/puppetproduction+1 -0
db1134: Disable notificationoperations/puppetproduction+1 -0
db1119: Disable notificationoperations/puppetproduction+1 -0
db1118: Disable notificationsoperations/puppetproduction+1 -0
db1164: Disable notificationsoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Apr 29 2022, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T08:58:58Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1163 (re)pooling @ 75%: Reboot T303171', diff saved to https://phabricator.wikimedia.org/P27006 and previous config saved to /var/cache/conftool/dbconfig/20220429-085858-kormat.json

Stashbot added a comment.Apr 29 2022, 9:14 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T09:14:02Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1163 (re)pooling @ 100%: Reboot T303171', diff saved to https://phabricator.wikimedia.org/P27007 and previous config saved to /var/cache/conftool/dbconfig/20220429-091401-kormat.json

Stashbot added a comment.Apr 29 2022, 9:36 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T09:36:07Z] <kormat@cumin1001> START - Cookbook sre.hosts.downtime for 1:30:00 on db1164.eqiad.wmnet with reason: Rebooting for T303171

Stashbot added a comment.Apr 29 2022, 9:36 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T09:36:12Z] <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:30:00 on db1164.eqiad.wmnet with reason: Rebooting for T303171

Stashbot added a comment.Apr 29 2022, 9:36 AM

Mentioned in SAL (#wikimedia-operations) [2022-04-29T09:36:18Z] <kormat@cumin1001> dbctl commit (dc=all): 'db1164 depooling: Rebooting for T303171', diff saved to https://phabricator.wikimedia.org/P27008 and previous config saved to /var/cache/conftool/dbconfig/20220429-093613-kormat.json

• Kormat updated the task description. (Show Details)Apr 29 2022, 9:36 AM
ops-monitoring-bot added a comment.Apr 29 2022, 9:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by kormat@cumin1001 for host db1164.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 29 2022, 10:06 AM

Cookbook cookbooks.sre.hosts.reimage started by kormat@cumin1001 for host db1164.eqiad.wmnet with OS bullseye executed with errors:

  • db1164 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
• Kormat mentioned this in T307198: db1164 fails to POST/boot/etc.Apr 29 2022, 10:27 AM
• Kormat added a comment.May 2 2022, 7:23 AM

Stalling until db1164 is back in service: T307198: db1164 fails to POST/boot/etc

• Kormat changed the task status from Open to Stalled.May 2 2022, 7:24 AM
• Cmjohnson closed subtask T307198: db1164 fails to POST/boot/etc as Resolved.May 11 2022, 7:19 PM
Marostegui changed the task status from Stalled to Open.May 12 2022, 1:33 PM

T307198 was fixed, I will reimage db1164 so it doesn't get behind for many more days

ops-monitoring-bot added a comment.May 12 2022, 1:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1164.eqiad.wmnet with OS bullseye

Ladsgroup added a subtask: T301312: Switchover s1 master (db1118 -> db1163).May 12 2022, 1:38 PM
gerritbot added a comment.May 12 2022, 1:40 PM

Change 791377 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1164: Disable notifications

https://gerrit.wikimedia.org/r/791377

gerritbot added a project: Patch-For-Review.May 12 2022, 1:40 PM

Change 791377 merged by Marostegui:

[operations/puppet@production] db1164: Disable notifications

https://gerrit.wikimedia.org/r/791377

Marostegui added a comment.May 12 2022, 2:00 PM

I have reimaged and started mysql again on db1164, it is 13 days behind the master :-/

Marostegui updated the task description. (Show Details)May 12 2022, 2:00 PM
ops-monitoring-bot added a comment.May 12 2022, 2:06 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1164.eqiad.wmnet with OS bullseye completed:

  • db1164 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205121334_marostegui_1255335_db1164.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.May 12 2022, 2:31 PM
Marostegui added a comment.May 15 2022, 6:29 AM

db1164 finally caught up, I will start repooling next week.

Ladsgroup closed subtask T301312: Switchover s1 master (db1118 -> db1163) as Resolved.May 19 2022, 6:19 AM
Marostegui updated the task description. (Show Details)May 20 2022, 6:37 AM
Jclark-ctr closed subtask T308246: db1164 power supply isn't redundant as Resolved.May 20 2022, 1:35 PM
gerritbot added a comment.May 23 2022, 4:58 AM

Change 796507 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db1118: Disable notifications

https://gerrit.wikimedia.org/r/796507

gerritbot added a project: Patch-For-Review.May 23 2022, 4:58 AM

Change 796507 merged by Marostegui:

[operations/puppet@production] db1118: Disable notifications

https://gerrit.wikimedia.org/r/796507

ops-monitoring-bot added a comment.May 23 2022, 5:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host db1118.eqiad.wmnet with OS bullseye

Marostegui updated the task description. (Show Details)May 23 2022, 5:22 AM
ops-monitoring-bot added a comment.May 23 2022, 5:35 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host db1118.eqiad.wmnet with OS bullseye completed:

  • db1118 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205230506_marostegui_978453_db1118.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Ladsgroup claimed this task.May 23 2022, 10:11 AM
Ladsgroup moved this task from Ready to In progress on the DBA board.
Ladsgroup added subscribers: • Kormat, Ladsgroup.

Kormat is out sick. I quickly finish this.

Stashbot added a comment.May 23 2022, 10:12 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T10:12:23Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1119 (T303171)', diff saved to https://phabricator.wikimedia.org/P28307 and previous config saved to /var/cache/conftool/dbconfig/20220523-101222-ladsgroup.json

gerritbot added a comment.May 23 2022, 10:14 AM

Change 797136 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db1119: Disable notification

https://gerrit.wikimedia.org/r/797136

gerritbot added a comment.May 23 2022, 10:15 AM

Change 797136 merged by Ladsgroup:

[operations/puppet@production] db1119: Disable notification

https://gerrit.wikimedia.org/r/797136

ops-monitoring-bot added a comment.May 23 2022, 10:24 AM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host db1119.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 23 2022, 10:51 AM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host db1119.eqiad.wmnet with OS bullseye completed:

  • db1119 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205231024_ladsgroup_1046863_db1119.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.May 23 2022, 11:00 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T11:00:44Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T303171)', diff saved to https://phabricator.wikimedia.org/P28312 and previous config saved to /var/cache/conftool/dbconfig/20220523-110043-ladsgroup.json

Ladsgroup updated the task description. (Show Details)May 23 2022, 11:04 AM
gerritbot added a comment.May 23 2022, 11:07 AM

Change 797174 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db1134: Disable notification

https://gerrit.wikimedia.org/r/797174

gerritbot added a comment.May 23 2022, 11:10 AM

Change 797174 merged by Ladsgroup:

[operations/puppet@production] db1134: Disable notification

https://gerrit.wikimedia.org/r/797174

Stashbot added a comment.May 23 2022, 11:46 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T11:45:59Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1119 (T303171)', diff saved to https://phabricator.wikimedia.org/P28317 and previous config saved to /var/cache/conftool/dbconfig/20220523-114559-ladsgroup.json

Stashbot added a comment.May 23 2022, 11:52 AM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T11:52:04Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1134 (T303171)', diff saved to https://phabricator.wikimedia.org/P28318 and previous config saved to /var/cache/conftool/dbconfig/20220523-115202-ladsgroup.json

ops-monitoring-bot added a comment.May 23 2022, 11:56 AM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host db1134.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 23 2022, 12:25 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host db1134.eqiad.wmnet with OS bullseye completed:

  • db1134 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205231156_ladsgroup_1063149_db1134.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Ladsgroup updated the task description. (Show Details)May 23 2022, 12:31 PM
Stashbot added a comment.May 23 2022, 12:39 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T12:39:44Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134 (T303171)', diff saved to https://phabricator.wikimedia.org/P28321 and previous config saved to /var/cache/conftool/dbconfig/20220523-123944-ladsgroup.json

gerritbot added a comment.May 23 2022, 12:44 PM

Change 797247 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db1135: Disable notification

https://gerrit.wikimedia.org/r/797247

gerritbot added a comment.May 23 2022, 12:45 PM

Change 797247 merged by Ladsgroup:

[operations/puppet@production] db1135: Disable notification

https://gerrit.wikimedia.org/r/797247

Stashbot added a comment.May 23 2022, 1:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T13:24:59Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1134 (T303171)', diff saved to https://phabricator.wikimedia.org/P28326 and previous config saved to /var/cache/conftool/dbconfig/20220523-132459-ladsgroup.json

Stashbot added a comment.May 23 2022, 1:32 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T13:32:30Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1135 (T303171)', diff saved to https://phabricator.wikimedia.org/P28328 and previous config saved to /var/cache/conftool/dbconfig/20220523-133228-ladsgroup.json

ops-monitoring-bot added a comment.May 23 2022, 1:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host db1135.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 23 2022, 2:14 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host db1135.eqiad.wmnet with OS bullseye completed:

  • db1135 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205231345_ladsgroup_1085481_db1135.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.May 23 2022, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T14:22:02Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T303171)', diff saved to https://phabricator.wikimedia.org/P28339 and previous config saved to /var/cache/conftool/dbconfig/20220523-142202-ladsgroup.json

Ladsgroup updated the task description. (Show Details)May 23 2022, 2:26 PM
gerritbot added a comment.May 23 2022, 2:29 PM

Change 797316 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db1184: Disable notification

https://gerrit.wikimedia.org/r/797316

gerritbot added a comment.May 23 2022, 2:32 PM

Change 797316 merged by Ladsgroup:

[operations/puppet@production] db1184: Disable notification

https://gerrit.wikimedia.org/r/797316

Stashbot added a comment.May 23 2022, 3:07 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T15:07:17Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1135 (T303171)', diff saved to https://phabricator.wikimedia.org/P28346 and previous config saved to /var/cache/conftool/dbconfig/20220523-150717-ladsgroup.json

Stashbot added a comment.May 23 2022, 3:12 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T15:12:09Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1184 (T303171)', diff saved to https://phabricator.wikimedia.org/P28348 and previous config saved to /var/cache/conftool/dbconfig/20220523-151207-ladsgroup.json

ops-monitoring-bot added a comment.May 23 2022, 3:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host db1184.eqiad.wmnet with OS bullseye

Ladsgroup updated the task description. (Show Details)May 23 2022, 3:41 PM
ops-monitoring-bot added a comment.May 23 2022, 3:49 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host db1184.eqiad.wmnet with OS bullseye completed:

  • db1184 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205231517_ladsgroup_1105614_db1184.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.May 23 2022, 4:01 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T16:01:06Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T303171)', diff saved to https://phabricator.wikimedia.org/P28353 and previous config saved to /var/cache/conftool/dbconfig/20220523-160105-ladsgroup.json

Stashbot added a comment.May 23 2022, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T16:46:21Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T303171)', diff saved to https://phabricator.wikimedia.org/P28357 and previous config saved to /var/cache/conftool/dbconfig/20220523-164621-ladsgroup.json

gerritbot added a comment.May 23 2022, 4:47 PM

Change 797344 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db1106: Disable notification

https://gerrit.wikimedia.org/r/797344

gerritbot added a comment.May 23 2022, 4:49 PM

Change 797344 merged by Ladsgroup:

[operations/puppet@production] db1106: Disable notification

https://gerrit.wikimedia.org/r/797344

Stashbot added a comment.May 23 2022, 4:50 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T16:50:50Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1106 (T303171)', diff saved to https://phabricator.wikimedia.org/P28358 and previous config saved to /var/cache/conftool/dbconfig/20220523-165045-ladsgroup.json

ops-monitoring-bot added a comment.May 23 2022, 4:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host db1106.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.May 23 2022, 5:26 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host db1106.eqiad.wmnet with OS bullseye completed:

  • db1106 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205231659_ladsgroup_1128779_db1106.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.May 23 2022, 5:34 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T17:34:39Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T303171)', diff saved to https://phabricator.wikimedia.org/P28359 and previous config saved to /var/cache/conftool/dbconfig/20220523-173439-ladsgroup.json

Ladsgroup removed a project: Patch-For-Review.May 23 2022, 6:12 PM
Ladsgroup updated the task description. (Show Details)
Stashbot added a comment.May 23 2022, 6:19 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-23T18:19:54Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1106 (T303171)', diff saved to https://phabricator.wikimedia.org/P28364 and previous config saved to /var/cache/conftool/dbconfig/20220523-181954-ladsgroup.json

Ladsgroup closed this task as Resolved.May 23 2022, 6:25 PM
Maintenance_bot moved this task from In progress to Done on the DBA board.May 23 2022, 6:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits