For capacity expansion of aux-k8s eqiad (please see rationale in parent task)
Specs intended to match existing workers
Networking Requirements: private vlan
Processor Requirements: 12 vCPU
Memory: 16 GB RAM
Disks: 50 GB
For capacity expansion of aux-k8s eqiad (please see rationale in parent task)
Specs intended to match existing workers
Networking Requirements: private vlan
Processor Requirements: 12 vCPU
Memory: 16 GB RAM
Disks: 50 GB
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T378742 aux-k8s: eqiad expansion, codfw creation, & future hopes and dreams | |||
| Resolved | herron | T378989 eqiad: (2x) aux-k8s-worker nodes |
LGTM! please use groups C and D if possible, that would give full diversity across ganeti groups
Change #1088359 had a related patch set uploaded (by Herron; author: Herron):
[operations/puppet@production] site: add insetup roles to aux-k8s-worker100[45]
Change #1088359 merged by Herron:
[operations/puppet@production] site: add insetup roles to aux-k8s-worker100[45]
Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm executed with errors:
ganeti1028:~# gnt-instance console aux-k8s-worker1004.eqiad.wmnet @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the RSA key sent by the remote host is SHA256:C9d5dCz/suTrJQqv6NtW3q/R241NXTC1GL3JMLfaMY4. Please contact your system administrator. Add correct host key in /dev/null to get rid of this message. Offending DSA key in /var/lib/ganeti/known_hosts:2 remove with: ssh-keygen -f "/var/lib/ganeti/known_hosts" -R "ganeti01.svc.eqiad.wmnet" RSA host key for ganeti01.svc.eqiad.wmnet has changed and you have requested strict checking. Host key verification failed. Failure: command execution error: Connection to console of instance aux-k8s-worker1004.eqiad.wmnet failed, please check cluster configuration
Looks like this VM got assigned to new ganeti host ganeti1045.eqiad.wmnet which might still be a work in progress T378921?
Learned that this is known issue T309724 and was able to work around it by using gnt-instance console --show-cmd (and turning off strict host key check)
Next issue at hand looks like:
2024-11-07 21:11:34,962 herron 986145 [ERROR] Exception raised while executing cookbook sre.hosts.reimage:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 251, in _run
raw_ret = runner.run()
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 711, in run
self._install_os()
File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 489, in _install_os
target = 'preseed/url=' if self.is_uefi else 'BOOT_IMAGE=debian-installer'
AttributeError: 'ReimageRunner' object has no attribute 'is_uefi'I think I see what is causing that, will upload a patch
Change #1088583 had a related patch set uploaded (by Herron; author: Herron):
[operations/cookbooks@master] reimage: don't check is_uefi on VMs
Change #1088583 merged by jenkins-bot:
[operations/cookbooks@master] reimage: don't check is_uefi on VMs
Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker1005.eqiad.wmnet with OS bookworm
Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker1005.eqiad.wmnet with OS bookworm completed:
Change #1088610 had a related patch set uploaded (by Herron; author: Herron):
[operations/puppet@production] aux-k8s: add 2 eqiad workers
Change #1091349 had a related patch set uploaded (by Herron; author: Herron):
[operations/puppet@production] aux_k8s: enable new eqiad workers
Change #1088610 merged by Herron:
[operations/puppet@production] role::aux_k8s::worker: add role to 2 new eqiad workers
Change #1091349 merged by Herron:
[operations/puppet@production] aux_k8s: enable new eqiad workers
I think we're good here!
NAME STATUS ROLES AGE VERSION aux-k8s-ctrl1002.eqiad.wmnet Ready control-plane 637d v1.23.14 aux-k8s-ctrl1003.eqiad.wmnet Ready control-plane 45d v1.23.14 aux-k8s-worker1002.eqiad.wmnet Ready <none> 637d v1.23.14 aux-k8s-worker1003.eqiad.wmnet Ready <none> 45d v1.23.14 aux-k8s-worker1004.eqiad.wmnet Ready <none> 3h1m v1.23.14 aux-k8s-worker1005.eqiad.wmnet Ready <none> 3h1m v1.23.14