⚓ T371132 Provision cookbook not setting serial console and other settings

Papaul created this task.Jul 26 2024, 6:23 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 26 2024, 6:23 PM

@Papaul is there an host that I can check via Redfish? In theory the change was a no-op, it should work as before (we just introduced the Supermicro support).

elukey triaged this task as Medium priority.Jul 29 2024, 2:32 PM

@elukey no we have no servers right now. We manually fixed al the db nodes where we were having isssue on.
https://phabricator.wikimedia.org/T369654. if we get any i will ping you

elukey added a project: User-Elukey.Jul 29 2024, 3:34 PM

@Papaul can you give us some provision cookbook run that didn't set it so we could check the logs please? Hostname and date/time if they were run multiple times.

@Volans all the server on https://phabricator.wikimedia.org/T369654

I've noticed that from the logs we're not setting any additional values since May 31st, from SAL I saw that we didn't had reimages between the 31st and when we merged the above patch.

I've commented on the patch on the code issue. I'm sorry I missed it during code review.

@Volans thank you

Papaul mentioned this in T369654: Q1:rack/setup/install db22[21-40].Jul 29 2024, 4:40 PM

Fix is in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1057927

Ideally we should re-run the provision on all hosts provisioned since then (list can be obtained in [1]), but we should first check if the missing changes could trigger a reboot of the underlying host or not, and it might also be host-dependent (some requiring reboot, some not).

[1] https://sal.toolforge.org/production?p=0&q=%22START+-+Cookbook+sre.hosts.provision%22&d=

I agree yes, there are probably some misconfigured hosts waiting there.. We can pick up one node that can be easily depooled and run the provision cookbook, I can check before/after changes with a script using Redfish.

Probably wikikube-worker1240?

elukey updated the task description. (Show Details)Jul 30 2024, 10:20 AM

elukey moved this task from Backlog to In Progress on the User-Elukey board.Jul 30 2024, 1:39 PM

elukey updated the task description. (Show Details)Jul 30 2024, 2:17 PM

Volans updated the task description. (Show Details)Jul 30 2024, 2:23 PM

elukey updated the task description. (Show Details)Jul 30 2024, 2:27 PM

From a databases point of view, only db1179 is in production but it is a slave, so it is easily depoolable. The other servers are not in production, they don't have any data.

Volans updated the task description. (Show Details)Jul 30 2024, 2:28 PM

Volans updated the task description. (Show Details)

pc1017 and pc2017 also belong to DBAs, they are not in production either. Can be done anytime

elukey updated the task description. (Show Details)Jul 30 2024, 2:31 PM

elukey updated the task description. (Show Details)Jul 30 2024, 4:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-07-31T06:47:53Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db1179 T371132', diff saved to https://phabricator.wikimedia.org/P67120 and previous config saved to /var/cache/conftool/dbconfig/20240731-064752-root.json