This server lost its linux bridges, and both neutron-linuxbridge-agent and nova-compute fail to operate.
Creating the bridges is something we do via puppet, and puppet is not failing in any way in the server.
I think I will reimage the server just to make sure our puppet code is sane.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | aborrero | T276964 cloudvirt2003-dev: neutron-linuxbridge-agent and nova-compute issues | |||
| Resolved | aborrero | T277014 cloudvirt2003-dev: debian installer partman recipe prompts for actions |
Script wmf-auto-reimage was launched by aborrero on cumin2001.codfw.wmnet for hosts:
cloudvirt2003-dev.codfw.wmnet
The log can be found in /var/log/wmf-auto-reimage/202103100907_aborrero_10408_cloudvirt2003-dev_codfw_wmnet.log.
Mentioned in SAL (#wikimedia-cloud) [2021-03-10T09:07:54Z] <arturo> [codfw1dev] reimaging cloudvirt2003-dev (T276964)
Completed auto-reimage of hosts:
['cloudvirt2003-dev.codfw.wmnet']
Of which those FAILED:
['cloudvirt2003-dev.codfw.wmnet']
The reimage was fine, but both nova-compute and neutron-linuxbridge-agent present issues when contacting the API services.
Mentioned in SAL (#wikimedia-cloud) [2021-03-10T11:53:55Z] <arturo> [codfw1dev] restart nova-conductor in all 3 cloudcontrol servers for T276964
Mentioned in SAL (#wikimedia-cloud) [2021-03-10T11:56:51Z] <arturo> [codfw1dev] restart rabbitmq-server in all 3 cloudcontrol servers for T276964
The server works nice now. I just double-checked.
This was probably fixed by @dcaro when upgraded python kombu & evenlet packages on codfw1dev. Thanks!