Page MenuHomePhabricator

Provision new RESTBase/Cassandra cluster nodes: restbase1028, restbase1029, restbase1030
Closed, ResolvedPublic

Description

Three new expansion hosts are ready to be provisioned in the RESTBase cluster (see: T241784: (Need by: TBD) rack/setup/install restbase1028, restbase1029, restbase1030).

  • restbase1028
  • restbase1029
  • restbase1030

Event Timeline

BPirkle added a subscriber: BPirkle.

Moving to Small Projects to clear the inbox. @Eevans and @hnowlan will likely collaborate on this task.

Which racks should these new hosts go onto? I am guessing one into each of a b and d, which are the eqiad racks atm (all at 4 hosts currently).

Which racks should these new hosts go onto? I am guessing one into each of a b and d, which are the eqiad racks atm (all at 4 hosts currently).

Correct.

Change 629723 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/dns@master] restbase102[89]/restbase1030: add cassandra hosts for new nodes

https://gerrit.wikimedia.org/r/629723

When where those hosts provisioned? Basically I'm asking when was the IP allocation script on Netbox run, because it explicitely supports this use case.

April it seems https://gerrit.wikimedia.org/r/c/operations/dns/+/592725/
I was wondering whether something odd had happened tbh but couldn't find suitable documentation for managing the other entries.

Ah so way before the IP allocation stuff was moved to Netbox. Let me add them to Netbox before merging.

Change 629723 merged by Hnowlan:
[operations/dns@master] restbase102[89]/restbase1030: add cassandra hosts for new nodes

https://gerrit.wikimedia.org/r/629723

Change 630106 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] restbase: add restbase102[89]/restbase1030

https://gerrit.wikimedia.org/r/630106

Change 630106 merged by Hnowlan:
[operations/puppet@production] restbase: add restbase102[89]/restbase1030

https://gerrit.wikimedia.org/r/630106

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['restbase1028.eqiad.wmnet', 'restbase1029.eqiad.wmnet', 'restbase1030.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009281608_hnowlan_381.log.

Completed auto-reimage of hosts:

['restbase1028.eqiad.wmnet', 'restbase1029.eqiad.wmnet', 'restbase1030.eqiad.wmnet']

and were ALL successful.

Change 630641 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] restbase: set role for new nodes

https://gerrit.wikimedia.org/r/630641

Change 630641 merged by Hnowlan:
[operations/puppet@production] restbase: set role for new nodes

https://gerrit.wikimedia.org/r/630641

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['restbase1028.eqiad.wmnet', 'restbase1029.eqiad.wmnet', 'restbase1030.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009290943_hnowlan_14117.log.

Completed auto-reimage of hosts:

['restbase1028.eqiad.wmnet', 'restbase1029.eqiad.wmnet', 'restbase1030.eqiad.wmnet']

Of which those FAILED:

['restbase1028.eqiad.wmnet', 'restbase1029.eqiad.wmnet', 'restbase1030.eqiad.wmnet']

Script wmf-auto-reimage was launched by hnowlan on cumin1001.eqiad.wmnet for hosts:

['restbase1028.eqiad.wmnet', 'restbase1029.eqiad.wmnet', 'restbase1030.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009291024_hnowlan_21992.log.

Change 632497 had a related patch set uploaded (by Hnowlan; owner: Hnowlan):
[operations/puppet@production] conftool-data: add new restbase hosts

https://gerrit.wikimedia.org/r/632497

Change 632497 merged by Hnowlan:
[operations/puppet@production] conftool-data: add new restbase hosts

https://gerrit.wikimedia.org/r/632497

hnowlan updated the task description. (Show Details)