Page MenuHomePhabricator
Create Task
Maniphest T352468

Provision new RESTBase cluster nodes: restbase20[28-35]
Closed, ResolvedPublic
Actions

Description

Provision new Cassandra hosts (as replacements/refreshes for restbase20[13-20]).

  • restbase2028
  • restbase2029
  • restbase2030
  • restbase2031
  • restbase2032
  • restbase2033
  • restbase2034
  • restbase2035

See also: T349758: Q1:rack/setup/install restbase20[28-35]

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Eevans created this task.Nov 30 2023, 7:01 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 30 2023, 7:01 PM
Eevans triaged this task as Medium priority.Nov 30 2023, 7:01 PM
Eevans mentioned this in T352469: Decommission restbase20[13-20]).Nov 30 2023, 7:09 PM
Eevans added a parent task: T352469: Decommission restbase20[13-20]).
ops-monitoring-bot added a comment.Nov 30 2023, 7:26 PM

Host rebooted by eevans@cumin1001 with reason: Apply new network settings

Eevans updated the task description. (Show Details)Nov 30 2023, 7:38 PM
gerritbot added a comment.Nov 30 2023, 8:07 PM

Change 979161 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2028

https://gerrit.wikimedia.org/r/979161

gerritbot added a project: Patch-For-Review.Nov 30 2023, 8:07 PM
gerritbot added a comment.Dec 4 2023, 9:28 PM

Change 979161 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2028

https://gerrit.wikimedia.org/r/979161

Maintenance_bot removed a project: Patch-For-Review.Dec 4 2023, 9:30 PM
gerritbot added a comment.Dec 4 2023, 10:15 PM

Change 980049 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: migrate restbase2028 to puppet 7

https://gerrit.wikimedia.org/r/980049

gerritbot added a project: Patch-For-Review.Dec 4 2023, 10:15 PM
gerritbot added a comment.Dec 4 2023, 11:51 PM

Change 980049 merged by Eevans:

[operations/puppet@production] restbase: migrate restbase2028 to puppet 7

https://gerrit.wikimedia.org/r/980049

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2023, 12:10 AM
Eevans updated the task description. (Show Details)Dec 6 2023, 3:04 PM
gerritbot added a comment.Dec 6 2023, 3:17 PM

Change 980884 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2029

https://gerrit.wikimedia.org/r/980884

gerritbot added a project: Patch-For-Review.Dec 6 2023, 3:17 PM
gerritbot added a comment.Dec 6 2023, 3:59 PM

Change 980884 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2029

https://gerrit.wikimedia.org/r/980884

Maintenance_bot removed a project: Patch-For-Review.Dec 6 2023, 4:11 PM
Stashbot added a comment.Dec 6 2023, 4:29 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-06T16:29:11Z] <urandom> bootstrapping Cassandra/restbase2020-a — T352468

Eevans updated the task description. (Show Details)Dec 7 2023, 5:19 PM
gerritbot added a comment.Dec 7 2023, 5:29 PM

Change 981371 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2030

https://gerrit.wikimedia.org/r/981371

gerritbot added a project: Patch-For-Review.Dec 7 2023, 5:29 PM
gerritbot added a comment.Dec 7 2023, 7:50 PM

Change 981371 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2030

https://gerrit.wikimedia.org/r/981371

Stashbot added a comment.Dec 7 2023, 8:05 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-07T20:05:51Z] <urandom> bootstrap Cassandra/restbase2030-a — T352468

Maintenance_bot removed a project: Patch-For-Review.Dec 7 2023, 8:18 PM
Eevans updated the task description. (Show Details)Dec 8 2023, 4:43 PM
gerritbot added a comment.Dec 8 2023, 7:57 PM

Change 981601 had a related patch set uploaded (by Eevans; author: Eevans):

[labs/private@master] keys & certs for missing restbase nodes

https://gerrit.wikimedia.org/r/981601

gerritbot added a project: Patch-For-Review.Dec 8 2023, 7:57 PM
gerritbot added a comment.Dec 8 2023, 8:53 PM

Change 981605 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2031

https://gerrit.wikimedia.org/r/981605

gerritbot added a comment.Dec 8 2023, 8:53 PM

Change 981606 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2032

https://gerrit.wikimedia.org/r/981606

gerritbot added a comment.Dec 8 2023, 8:53 PM

Change 981607 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2033

https://gerrit.wikimedia.org/r/981607

gerritbot added a comment.Dec 8 2023, 8:53 PM

Change 981608 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2034

https://gerrit.wikimedia.org/r/981608

gerritbot added a comment.Dec 8 2023, 8:53 PM

Change 981609 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2035

https://gerrit.wikimedia.org/r/981609

gerritbot added a comment.Dec 11 2023, 6:52 PM

Change 981605 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2031

https://gerrit.wikimedia.org/r/981605

Eevans mentioned this in T353189: Issue(s) bootstrapping Cassandra nodes.Dec 11 2023, 7:55 PM
Eevans updated the task description. (Show Details)Dec 13 2023, 12:03 AM
gerritbot added a comment.Dec 13 2023, 7:35 PM

Change 981606 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2032

https://gerrit.wikimedia.org/r/981606

Eevans updated the task description. (Show Details)Dec 14 2023, 3:06 PM
Eevans updated the task description. (Show Details)Dec 14 2023, 7:22 PM

restbase2032 was erroneously bootstrapped into row B (should be row C), and will have to be decommissioned (and re-bootstrapped). :(

gerritbot added a comment.Dec 15 2023, 3:01 PM

Change 983431 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase2032: fix erroneous row

https://gerrit.wikimedia.org/r/983431

gerritbot added a comment.Dec 15 2023, 3:02 PM

Change 983431 merged by Eevans:

[operations/puppet@production] restbase2032: fix erroneous row

https://gerrit.wikimedia.org/r/983431

Eevans moved this task from Backlog to In-Progress on the Cassandra board.Dec 18 2023, 7:56 PM
gerritbot added a comment.Dec 19 2023, 2:38 PM

Change 984204 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/puppet@production] Fix insetup role for restbase203[3-5]

https://gerrit.wikimedia.org/r/984204

gerritbot added a comment.Dec 19 2023, 2:46 PM

Change 984204 merged by Alexandros Kosiaris:

[operations/puppet@production] Fix insetup role for restbase203[3-5]

https://gerrit.wikimedia.org/r/984204

Eevans mentioned this in T353456: Cassandra/restbase2029-a (and others) oom-killed (kernel).Dec 19 2023, 7:35 PM
gerritbot added a comment.Dec 20 2023, 7:24 PM

Change 981607 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2033

https://gerrit.wikimedia.org/r/981607

Eevans updated the task description. (Show Details)Dec 20 2023, 7:29 PM
ops-monitoring-bot added a comment.Dec 20 2023, 7:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 8:30 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye executed with errors:

  • restbase2033 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312201951_eevans_1186823_restbase2033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 20 2023, 8:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 8:47 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye executed with errors:

  • restbase2033 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 20 2023, 8:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 9:26 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye executed with errors:

  • restbase2033 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202108_eevans_1197116_restbase2033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 20 2023, 9:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 10:03 PM

Cookbook cookbooks.sre.hosts.reimage started by eevans@cumin1002 for host restbase2033.codfw.wmnet with OS bullseye completed:

  • restbase2033 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202148_eevans_1204161_restbase2033.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Dec 20 2023, 10:40 PM

Change 984647 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set production role and add config for restbase2033

https://gerrit.wikimedia.org/r/984647

gerritbot added a comment.Dec 21 2023, 2:24 PM

Change 984647 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2033

https://gerrit.wikimedia.org/r/984647

Eevans updated the task description. (Show Details)Dec 22 2023, 2:58 PM
gerritbot added a comment.Jan 4 2024, 8:32 PM

Change 981608 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2034

https://gerrit.wikimedia.org/r/981608

Eevans updated the task description. (Show Details)Jan 5 2024, 7:21 PM
gerritbot added a comment.Jan 5 2024, 7:32 PM

Change 981601 merged by Eevans:

[labs/private@master] restbase: add missing keys & certs, remove obsolete

https://gerrit.wikimedia.org/r/981601

Eevans mentioned this in rLPRI3c8827944a8b: restbase: add missing keys & certs, remove obsolete.Jan 5 2024, 7:51 PM
gerritbot added a comment.Jan 8 2024, 3:49 PM

Change 981609 merged by Eevans:

[operations/puppet@production] restbase: set production role and add config for restbase2035

https://gerrit.wikimedia.org/r/981609

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2024, 4:32 PM
Eevans updated the task description. (Show Details)Jan 9 2024, 2:19 PM
gerritbot added a comment.Jan 9 2024, 9:02 PM

Change 989248 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: configure new hosts for partition reuse

https://gerrit.wikimedia.org/r/989248

gerritbot added a project: Patch-For-Review.Jan 9 2024, 9:02 PM
gerritbot added a comment.Jan 10 2024, 3:23 PM

Change 989248 merged by Eevans:

[operations/puppet@production] restbase: configure new hosts for partition reuse

https://gerrit.wikimedia.org/r/989248

Maintenance_bot removed a project: Patch-For-Review.Jan 10 2024, 3:31 PM
Eevans closed this task as Resolved.Feb 4 2024, 1:48 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits