Page MenuHomePhabricator

Upload redfish licenses to supermicro hosts
Closed, ResolvedPublic

Description

Hosts with the license uploaded:

  • an-conf1004 (re-run the provision cookbook to get the canonical config)
  • an-conf1005 (re-run the provision cookbook to get the canonical config)
  • an-conf1006 (re-run the provision cookbook to get the canonical config)
  • dbproxy1028 (re-run the provision cookbook to get the canonical config)
  • dbproxy1029 (re-run the provision cookbook to get the canonical config)
  • dbproxy2005 (re-run the provision cookbook to get the canonical config)
  • dbproxy2006 (re-run the provision cookbook to get the canonical config)
  • dbproxy2007 (re-run the provision cookbook to get the canonical config)
  • dbproxy2008 (re-run the provision cookbook to get the canonical config)
  • krb1002 (re-run the provision cookbook to get the canonical config)
  • deploy1003 (re-run the provision cookbook to get the canonical config)
  • parsoidtest1001 (re-run the provision cookbook to get the canonical config)
  • ml-serve2009 (re-run the provision cookbook to get the canonical config)
  • ml-serve2010 (re-run the provision cookbook to get the canonical config)
  • ml-serve2011 (re-run the provision cookbook to get the canonical config)
  • ml-staging2003 (re-run the provision cookbook to get the canonical config)
  • ganeti2035 (re-run the provision cookbook to get the canonical config)
  • ganeti2036 (re-run the provision cookbook to get the canonical config)
  • ganeti2037 (re-run the provision cookbook to get the canonical config)
  • ganeti2038 (re-run the provision cookbook to get the canonical config)
  • ganeti2039 (re-run the provision cookbook to get the canonical config)
  • ganeti2040 (re-run the provision cookbook to get the canonical config)
  • ganeti2041 (re-run the provision cookbook to get the canonical config)
  • ganeti2042 (re-run the provision cookbook to get the canonical config)
  • ganeti2043 (re-run the provision cookbook to get the canonical config)
  • ganeti2044 (re-run the provision cookbook to get the canonical config)
  • sretest2001 (re-run the provision cookbook to get the canonical config)
  • ml-serve1009 (re-run the provision cookbook to get the canonical config)
  • ml-serve1010 (re-run the provision cookbook to get the canonical config)
  • ml-serve1011 (re-run the provision cookbook to get the canonical config)
  • ml-lab1001 (re-run the provision cookbook to get the canonical config)
  • ml-lab1002 (re-run the provision cookbook to get the canonical config)
  • dse-k8s-worker1009 (re-run the provision cookbook to get the canonical config)
  • ganeti1039 (re-run the provision cookbook to get the canonical config)
  • ganeti1040 (re-run the provision cookbook to get the canonical config)
  • ganeti1041 (re-run the provision cookbook to get the canonical config)
  • ganeti1042 (re-run the provision cookbook to get the canonical config)
  • ganeti1043 (re-run the provision cookbook to get the canonical config)
  • ganeti1044 (re-run the provision cookbook to get the canonical config)
  • ganeti1045 (re-run the provision cookbook to get the canonical config)
  • ganeti1046 (re-run the provision cookbook to get the canonical config)
  • ganeti1047 (re-run the provision cookbook to get the canonical config)
  • ganeti1048 (re-run the provision cookbook to get the canonical config)
  • ganeti1049 (re-run the provision cookbook to get the canonical config)
  • ganeti1050 (re-run the provision cookbook to get the canonical config)
  • ganeti1051 (re-run the provision cookbook to get the canonical config)
  • ganeti1052 (re-run the provision cookbook to get the canonical config)

Procedure:

  • Get the serial from the mac address of the target host, since Supermicro's license list comes indexed by mac address. Use this spreadsheet.
  • Get the hostname related to the serial using this API.
test-cookbook -c 1076975 sre.hosts.provision --no-users --no-dhcp --only-redfish-license $hostname

After the above, getting the Bios settings should work via Redfish.

Event Timeline

elukey triaged this task as Medium priority.Oct 14 2024, 2:40 PM

All the licenses are applied, the last steps are to run the provision cookbook on all nodes.

elukey claimed this task.

Finally all the hosts without the license, that were manually configured, should be ok. The only remaining thing left is to make a quick pass and set the ADMIN password equal to the root one, since I've run provision with --no-users and --no-dhcp to avoid a change of status in Netbox (Active hosts cannot get provisioned without -no-dhcp and --no-users for safety).

I've used the following script (added to the spicerack_shell.py one) to bulk-update the accounts:

from spicerack.interactive import get_management_password
mgmt_password = get_management_password()
hosts = ["an-conf1004","an-conf1005","an-conf1006","dbproxy1028","dbproxy1029","dbproxy2005","dbproxy2006","dbproxy2007","dbproxy2008","krb1002","deploy1003","parsoidtest1001","ml-serve2009","ml-serve2010","ml-serve2011","ml-staging2003","ganeti2035","ganeti2036","ganeti2037","ganeti2038","ganeti2039","ganeti2040","ganeti2041","ganeti2042","ganeti2043","ganeti2044","sretest2001","ml-serve1009","ml-serve1010","ml-serve1011","ml-lab1001","ml-lab1002","dse-k8s-worker1009","ganeti1039","ganeti1040","ganeti1041","ganeti1042","ganeti1043"]

for host in hosts:
    print(host)
    r = spicerack.redfish(host, password = mgmt_password)
    r.change_user_password('ADMIN', mgmt_password)

Tested a couple of nodes and all worked as expected. At this point we should be able to close!

ETOOSOON :)

It seems that ganeti1044+ hosts were already provisioned, and I didn't notice an error when uploading the license to 1044:

Error while configuring the Redfish license: POST https://10.65.4.77/redfish/v1/Managers/1/LicenseManager/Actions/LicenseManager.ActivateLicense returned HTTP 400

The other hosts seem fine, re-running provision on all of them.

Run provision on ganeti1045+, and fixed the ADMIN password as well.

Last step is to figure out why ganeti1044's license doesn't work, I contacted Supermicro about it.

The mac address that Supermicro provided to us on the server's label is not correct, the last digit that we have is 8 meanwhile the MAC address returned by Redfish lists B, and the mac address is used to generate the license (I am 99% sure of it). Followed up with Supermicro via email.

elukey updated the task description. (Show Details)

Supermicro sent a new license for 1044 that worked, and I've ran successfully the provision cookbook.