Creating this ticket to track work required to bring wqds hosts wdqs101[4,5,6] into service. These hosts are replacing wdqs100[3-5]; as such one of these hosts (wdqs1016) will be in wdqs-internal to match wdqs1003's old role.
AC
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | None | T294806 [Epic] Search platform - Hardware requests for 2021-2022 | |||
| Unknown Object (Task) | |||||
| Resolved | • Cmjohnson | T307138 Q4:(Need By: TBD) rack/setup/install wdqs101[4,5,6] | |||
| Resolved | bking | T314890 Service implementation for wdqs101[4,5,6] |
Change 821785 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] wdqs: bring more hosts online
Change 821785 merged by Bking:
[operations/puppet@production] wdqs: bring more hosts online
Mentioned in SAL (#wikimedia-operations) [2022-08-09T19:55:27Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: T314890
Mentioned in SAL (#wikimedia-operations) [2022-08-09T19:55:51Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on wdqs1015.eqiad.wmnet with reason: T314890
Mentioned in SAL (#wikimedia-operations) [2022-08-09T19:56:26Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on wdqs1016.eqiad.wmnet with reason: T314890
Mentioned in SAL (#wikimedia-operations) [2022-08-09T19:56:40Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on wdqs1016.eqiad.wmnet with reason: T314890
Mentioned in SAL (#wikimedia-operations) [2022-08-09T19:57:06Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on wdqs1014.eqiad.wmnet with reason: T314890
Mentioned in SAL (#wikimedia-operations) [2022-08-09T19:57:12Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on wdqs1014.eqiad.wmnet with reason: T314890
On new hosts only, the data transfer cookbook fails on the repooling step. I believe this is because the newly-provisioned server is not yet enabled in the load balancer pool.
To enable it, run the following command from cumin:
confctl select name=wdqs1014.eqiad.wmnet set/weight=10:pooled=yes
Mentioned in SAL (#wikimedia-operations) [2022-08-22T13:37:47Z] <bking@cumin1001> START - Cookbook sre.hosts.downtime for 3:00:00 on wdqs[1014-1016].eqiad.wmnet with reason: T314890
Mentioned in SAL (#wikimedia-operations) [2022-08-22T13:38:02Z] <bking@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on wdqs[1014-1016].eqiad.wmnet with reason: T314890
We made a slight mistake: one of these hosts needed to be in wdqs-internal since wdqs1003 (one of the hosts replaced by these new hosts) is.
Re-opening this and updating the task description's AC accordingly.
Steps left to do:
Oops, meant this to be in progress. Changing status to Open from previous state of Resolved.
Change 955396 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] wdqs-internal: switch wdqs1016 from public to internal role
Host was reimaged but patch wasn't yet merged. Merging patch and rolling the re-image again.
Change 955396 merged by Ryan Kemper:
[operations/puppet@production] wdqs-internal: switch wdqs1016 from public to internal role
Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin1001 for host wdqs1016.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin1001 for host wdqs1016.eqiad.wmnet with OS bullseye completed:
{"wdqs1016.eqiad.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=eqiad,cluster=wdqs-internal,service=wdqs"}