Page MenuHomePhabricator

eqiad: (2x) aux-k8s-worker nodes
Closed, ResolvedPublic

Description

For capacity expansion of aux-k8s eqiad (please see rationale in parent task)

Specs intended to match existing workers
Networking Requirements: private vlan
Processor Requirements: 12 vCPU
Memory: 16 GB RAM
Disks: 50 GB

Event Timeline

LGTM! please use groups C and D if possible, that would give full diversity across ganeti groups

Change #1088359 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] site: add insetup roles to aux-k8s-worker100[45]

https://gerrit.wikimedia.org/r/1088359

Change #1088359 merged by Herron:

[operations/puppet@production] site: add insetup roles to aux-k8s-worker100[45]

https://gerrit.wikimedia.org/r/1088359

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker1004.eqiad.wmnet with OS bookworm executed with errors:

  • aux-k8s-worker1004 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console aux-k8s-worker1004.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ganeti1028:~# gnt-instance console aux-k8s-worker1004.eqiad.wmnet
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:C9d5dCz/suTrJQqv6NtW3q/R241NXTC1GL3JMLfaMY4.
Please contact your system administrator.
Add correct host key in /dev/null to get rid of this message.
Offending DSA key in /var/lib/ganeti/known_hosts:2
  remove with:
  ssh-keygen -f "/var/lib/ganeti/known_hosts" -R "ganeti01.svc.eqiad.wmnet"
RSA host key for ganeti01.svc.eqiad.wmnet has changed and you have requested strict checking.
Host key verification failed.
Failure: command execution error:
Connection to console of instance aux-k8s-worker1004.eqiad.wmnet failed, please check cluster configuration

Looks like this VM got assigned to new ganeti host ganeti1045.eqiad.wmnet which might still be a work in progress T378921?

ganeti1028:~# gnt-instance console aux-k8s-worker1004.eqiad.wmnet
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:C9d5dCz/suTrJQqv6NtW3q/R241NXTC1GL3JMLfaMY4.
Please contact your system administrator.
Add correct host key in /dev/null to get rid of this message.
Offending DSA key in /var/lib/ganeti/known_hosts:2
  remove with:
  ssh-keygen -f "/var/lib/ganeti/known_hosts" -R "ganeti01.svc.eqiad.wmnet"
RSA host key for ganeti01.svc.eqiad.wmnet has changed and you have requested strict checking.
Host key verification failed.
Failure: command execution error:
Connection to console of instance aux-k8s-worker1004.eqiad.wmnet failed, please check cluster configuration

Looks like this VM got assigned to new ganeti host ganeti1045.eqiad.wmnet which might still be a work in progress T378921?

Learned that this is known issue T309724 and was able to work around it by using gnt-instance console --show-cmd (and turning off strict host key check)

Next issue at hand looks like:

2024-11-07 21:11:34,962 herron 986145 [ERROR] Exception raised while executing cookbook sre.hosts.reimage:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 251, in _run
    raw_ret = runner.run()
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 711, in run
    self._install_os()
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 489, in _install_os
    target = 'preseed/url=' if self.is_uefi else 'BOOT_IMAGE=debian-installer'
AttributeError: 'ReimageRunner' object has no attribute 'is_uefi'

I think I see what is causing that, will upload a patch

Change #1088583 had a related patch set uploaded (by Herron; author: Herron):

[operations/cookbooks@master] reimage: don't check is_uefi on VMs

https://gerrit.wikimedia.org/r/1088583

Change #1088583 merged by jenkins-bot:

[operations/cookbooks@master] reimage: don't check is_uefi on VMs

https://gerrit.wikimedia.org/r/1088583

Cookbook cookbooks.sre.hosts.reimage was started by herron@cumin1002 for host aux-k8s-worker1005.eqiad.wmnet with OS bookworm

Cookbook cookbooks.sre.hosts.reimage started by herron@cumin1002 for host aux-k8s-worker1005.eqiad.wmnet with OS bookworm completed:

  • aux-k8s-worker1005 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411081737_herron_1149095_aux-k8s-worker1005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change #1088610 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] aux-k8s: add 2 eqiad workers

https://gerrit.wikimedia.org/r/1088610

Change #1091349 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] aux_k8s: enable new eqiad workers

https://gerrit.wikimedia.org/r/1091349

Change #1088610 merged by Herron:

[operations/puppet@production] role::aux_k8s::worker: add role to 2 new eqiad workers

https://gerrit.wikimedia.org/r/1088610

Change #1091349 merged by Herron:

[operations/puppet@production] aux_k8s: enable new eqiad workers

https://gerrit.wikimedia.org/r/1091349

I think we're good here!

NAME                             STATUS   ROLES           AGE    VERSION
aux-k8s-ctrl1002.eqiad.wmnet     Ready    control-plane   637d   v1.23.14
aux-k8s-ctrl1003.eqiad.wmnet     Ready    control-plane   45d    v1.23.14
aux-k8s-worker1002.eqiad.wmnet   Ready    <none>          637d   v1.23.14
aux-k8s-worker1003.eqiad.wmnet   Ready    <none>          45d    v1.23.14
aux-k8s-worker1004.eqiad.wmnet   Ready    <none>          3h1m   v1.23.14
aux-k8s-worker1005.eqiad.wmnet   Ready    <none>          3h1m   v1.23.14