Page MenuHomePhabricator
Create Task
Maniphest T300006

Upgrade es5 to Bullseye
Closed, ResolvedPublic
Actions

Description

Let's upgrade es4 to Bullseye.
es5 is RW not RO, so it does require a proper DB switchover.

  • es2025
  • es2024
  • es2023
  • es1025
  • es1024
  • es1023

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+1 -0es1024: Disable notifications
operations/puppetproduction+1 -0es1023: Disable notifications
operations/puppetproduction+1 -0es1025: Disable notifications
operations/puppetproduction+1 -0es2024: Disable notifications
operations/puppetproduction+2 -0es2025: Disable notifications
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Jan 25 2022, 9:55 AM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJan 25 2022, 9:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• Marostegui renamed this task from Upgrade es4 to Bullseye to Upgrade es5 to Bullseye.Jan 25 2022, 9:55 AM
• Marostegui triaged this task as Medium priority.
• Marostegui moved this task from Triage to Ready on the DBA board.
Stashbot added a comment.Jan 26 2022, 10:25 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-26T10:25:45Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es2025 (T300006)', diff saved to https://phabricator.wikimedia.org/P19264 and previous config saved to /var/cache/conftool/dbconfig/20220126-102445-ladsgroup.json

gerritbot added a comment.Jan 26 2022, 10:25 AM

Change 757396 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es2025: Disable notifications

https://gerrit.wikimedia.org/r/757396

gerritbot added a project: Patch-For-Review.Jan 26 2022, 10:25 AM
Ladsgroup moved this task from Ready to In progress on the DBA board.Jan 26 2022, 10:26 AM
gerritbot added a comment.Jan 26 2022, 10:26 AM

Change 757396 merged by Ladsgroup:

[operations/puppet@production] es2025: Disable notifications

https://gerrit.wikimedia.org/r/757396

ops-monitoring-bot added a comment.Jan 26 2022, 10:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es2025.codfw.wmnet with OS bullseye

jcrespo added a subscriber: jcrespo.Jan 26 2022, 10:54 AM
ops-monitoring-bot added a comment.Jan 26 2022, 11:04 AM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es2025.codfw.wmnet with OS bullseye completed:

  • es2025 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201261028_ladsgroup_3126_es2025.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.Jan 26 2022, 11:10 AM
Stashbot added a comment.Jan 26 2022, 11:24 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-26T11:24:39Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es2025 (T300006)', diff saved to https://phabricator.wikimedia.org/P19280 and previous config saved to /var/cache/conftool/dbconfig/20220126-112439-ladsgroup.json

Stashbot added a comment.Jan 26 2022, 11:27 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-26T11:27:20Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es2025 (T300006)', diff saved to https://phabricator.wikimedia.org/P19281 and previous config saved to /var/cache/conftool/dbconfig/20220126-112719-ladsgroup.json

Ladsgroup updated the task description. (Show Details)Jan 26 2022, 11:30 AM
gerritbot added a comment.Jan 26 2022, 11:32 AM

Change 757417 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es2024: Disable notifications

https://gerrit.wikimedia.org/r/757417

gerritbot added a project: Patch-For-Review.Jan 26 2022, 11:32 AM
gerritbot added a comment.Jan 26 2022, 11:35 AM

Change 757417 merged by Ladsgroup:

[operations/puppet@production] es2024: Disable notifications

https://gerrit.wikimedia.org/r/757417

Stashbot added a comment.Jan 26 2022, 11:36 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-26T11:36:26Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es2024 (T300006)', diff saved to https://phabricator.wikimedia.org/P19284 and previous config saved to /var/cache/conftool/dbconfig/20220126-113626-ladsgroup.json

ops-monitoring-bot added a comment.Jan 26 2022, 11:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es2024.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Jan 26 2022, 12:11 PM
ops-monitoring-bot added a comment.Jan 26 2022, 12:18 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es2024.codfw.wmnet with OS bullseye completed:

  • es2024 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201261141_ladsgroup_29141_es2024.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 26 2022, 12:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-26T12:38:40Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es2024 (T300006)', diff saved to https://phabricator.wikimedia.org/P19301 and previous config saved to /var/cache/conftool/dbconfig/20220126-123839-ladsgroup.json

Ladsgroup updated the task description. (Show Details)Jan 26 2022, 12:41 PM
gerritbot added a comment.Jan 26 2022, 1:55 PM

Change 757446 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1025: Disable notifications

https://gerrit.wikimedia.org/r/757446

gerritbot added a project: Patch-For-Review.Jan 26 2022, 1:55 PM
gerritbot added a comment.Jan 26 2022, 1:58 PM

Change 757446 merged by Marostegui:

[operations/puppet@production] es1025: Disable notifications

https://gerrit.wikimedia.org/r/757446

ops-monitoring-bot added a comment.Jan 26 2022, 3:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1025.eqiad.wmnet with OS bullseye

• Marostegui updated the task description. (Show Details)Jan 26 2022, 3:38 PM
ops-monitoring-bot added a comment.Jan 26 2022, 3:40 PM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1025.eqiad.wmnet with OS bullseye completed:

  • es1025 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201261507_marostegui_30725_es1025.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 27 2022, 10:47 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T10:47:36Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on es[2024-2025].codfw.wmnet with reason: Reimage of the master T300006

Stashbot added a comment.Jan 27 2022, 10:47 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T10:47:41Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on es[2024-2025].codfw.wmnet with reason: Reimage of the master T300006

ops-monitoring-bot added a comment.Jan 27 2022, 10:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es2023.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 27 2022, 11:29 AM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es2023.codfw.wmnet with OS bullseye completed:

  • es2023 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201271050_ladsgroup_17459_es2023.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Ladsgroup updated the task description. (Show Details)Jan 27 2022, 11:56 AM
gerritbot added a comment.Jan 27 2022, 12:04 PM

Change 757638 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es1023: Disable notifications

https://gerrit.wikimedia.org/r/757638

gerritbot added a comment.Jan 27 2022, 12:05 PM

Change 757638 merged by Ladsgroup:

[operations/puppet@production] es1023: Disable notifications

https://gerrit.wikimedia.org/r/757638

Stashbot added a comment.Jan 27 2022, 12:06 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T12:06:49Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es1023 (T300006)', diff saved to https://phabricator.wikimedia.org/P19444 and previous config saved to /var/cache/conftool/dbconfig/20220127-120648-ladsgroup.json

ops-monitoring-bot added a comment.Jan 27 2022, 1:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es1023.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 27 2022, 2:16 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es1023.eqiad.wmnet with OS bullseye completed:

  • es1023 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201271345_ladsgroup_2000_es1023.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 27 2022, 2:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T14:25:18Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023 (T300006)', diff saved to https://phabricator.wikimedia.org/P19475 and previous config saved to /var/cache/conftool/dbconfig/20220127-142517-ladsgroup.json

Stashbot added a comment.Jan 27 2022, 3:10 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T15:10:32Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1023 (T300006)', diff saved to https://phabricator.wikimedia.org/P19483 and previous config saved to /var/cache/conftool/dbconfig/20220127-151032-ladsgroup.json

Ladsgroup updated the task description. (Show Details)Jan 27 2022, 5:34 PM
Ladsgroup added a comment.Jan 27 2022, 5:36 PM

Now only the master left, we will do that maybe next week?

• Marostegui added a comment.Jan 27 2022, 6:32 PM
In T300006#7657115, @Ladsgroup wrote:

Now only the master left, we will do that maybe next week?

Take a look at the very same thing I have to do for es4: T300127

• Marostegui moved this task from In progress to Blocked on the DBA board.Feb 1 2022, 2:14 PM
Ladsgroup mentioned this in T300976: Switchover es5 master.Feb 4 2022, 1:46 PM
Stashbot added a comment.Feb 15 2022, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T09:49:31Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T300006

Stashbot added a comment.Feb 15 2022, 9:49 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T09:49:36Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T300006

Stashbot added a comment.Feb 15 2022, 10:02 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T10:02:53Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Set es1023 with weight 0 T300006', diff saved to https://phabricator.wikimedia.org/P20772 and previous config saved to /var/cache/conftool/dbconfig/20220215-100253-ladsgroup.json

Stashbot added a comment.Feb 15 2022, 10:10 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T10:10:21Z] <Amir1> Starting es5 eqiad failover from es1024 to es1023 - T300006

Stashbot added a comment.Feb 15 2022, 10:14 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T10:14:13Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Promote es1023 to es5 primary and set section read-write T300006', diff saved to https://phabricator.wikimedia.org/P20776 and previous config saved to /var/cache/conftool/dbconfig/20220215-101412-root.json

Stashbot added a comment.Feb 15 2022, 10:18 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T10:18:18Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Setting weight to es1023 T300006', diff saved to https://phabricator.wikimedia.org/P20777 and previous config saved to /var/cache/conftool/dbconfig/20220215-101817-root.json

Ladsgroup closed subtask T300976: Switchover es5 master as Resolved.Feb 15 2022, 10:38 AM
gerritbot added a comment.Feb 15 2022, 10:53 AM

Change 762783 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] es1024: Disable notifications

https://gerrit.wikimedia.org/r/762783

gerritbot added a comment.Feb 15 2022, 10:57 AM

Change 762783 merged by Ladsgroup:

[operations/puppet@production] es1024: Disable notifications

https://gerrit.wikimedia.org/r/762783

Stashbot added a comment.Feb 15 2022, 11:04 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T11:04:20Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling es1024 (T300006)', diff saved to https://phabricator.wikimedia.org/P20781 and previous config saved to /var/cache/conftool/dbconfig/20220215-110420-ladsgroup.json

ops-monitoring-bot added a comment.Feb 15 2022, 3:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by ladsgroup@cumin1001 for host es1024.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 15 2022, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage started by ladsgroup@cumin1001 for host es1024.eqiad.wmnet with OS bullseye completed:

  • es1024 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202151516_ladsgroup_21612_es1024.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 15 2022, 4:00 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T16:00:55Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1024 (T300006)', diff saved to https://phabricator.wikimedia.org/P20813 and previous config saved to /var/cache/conftool/dbconfig/20220215-160055-ladsgroup.json

Stashbot added a comment.Feb 15 2022, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-15T16:46:11Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance es1024 (T300006)', diff saved to https://phabricator.wikimedia.org/P20819 and previous config saved to /var/cache/conftool/dbconfig/20220215-164611-ladsgroup.json

Ladsgroup closed this task as Resolved.Feb 15 2022, 6:19 PM
Ladsgroup updated the task description. (Show Details)
Ladsgroup moved this task from Blocked to Done on the DBA board.
Maintenance_bot moved this task from Incoming to Done on the User-Ladsgroup board.Feb 15 2022, 7:15 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL