openstack galera no recent writes 2025-03-04, suspected network hardware problem
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• aborrero
	Mar 4 2025, 8:46 AM

Description

We got this alert today:

Checking with:

aborrero@cloudcontrol1005:~ $ sudo mysql -u root
MariaDB [(none)]> SHOW STATUS LIKE "wsrep_last_committed";
+----------------------+------------+
| Variable_name        | Value      |
+----------------------+------------+
| wsrep_last_committed | 1070411385 |
+----------------------+------------+
1 row in set (0.001 sec)

Seems only cloudcontrol1005 is a bit behind:

cloudcontrol1005 | wsrep_last_committed | 1070411385 |
cloudcontrol1006 | wsrep_last_committed | 1070411666 |
cloudcontrol1007 | wsrep_last_committed | 1070411666 |

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	inventory: place cloudcontrol1005 last on the inventory	cloud/wmcs-cookbooks	main	+2 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Andrew	T387946 SystemdUnitDown
Resolved	VRiley-WMF	T387828 openstack galera no recent writes 2025-03-04, suspected network hardware problem
Resolved	• aborrero	T387844 NodeDown Node cloudcontrol1005 has been down for long.
Resolved	dcaro	T387845 [maintain-dbusers] move to another cloudcontrol node

Event Timeline

• aborrero created this task.Mar 4 2025, 8:46 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 4 2025, 8:46 AM

Mentioned in SAL (#wikimedia-cloud) [2025-03-04T08:52:04Z] <arturo> T387828 depooled galera on cloudcontrol1005

Host rebooted by aborrero@cumin1002 with reason: galera problem

• aborrero changed the task status from Open to In Progress.Mar 4 2025, 8:59 AM

• aborrero triaged this task as High priority.

• aborrero moved this task from Backlog to Doing on the User-aborrero board.

server wont shutdown for reboot, so we are force-rebooting it

when force-rebooted, the server did not have the correct network configuration

the switch port the server is connected to is somehow down:

Not sure if there is a problem in the switch port, the cable or similar.

• aborrero renamed this task from openstack galera no recent writes 2025-03-04 to openstack galera no recent writes 2025-03-04, suspected network hardware problem.Mar 4 2025, 9:48 AM

Hey @VRiley-WMF and/or @Jclark-ctr, Could you please check on-site if there is a loose cable in either the switch port or in the server NIC for cloudcontrol1005?

Server: https://netbox.wikimedia.org/dcim/devices/2613/
Switch port: https://netbox.wikimedia.org/dcim/interfaces/30016/

thanks!

• aborrero assigned this task to VRiley-WMF.Mar 4 2025, 9:51 AM

• aborrero moved this task from Doing to Blocked/waiting on the User-aborrero board.Mar 4 2025, 10:29 AM

Maintenance_bot added a project: SRE.Mar 4 2025, 10:29 AM

• aborrero added a subtask: T387844: NodeDown Node cloudcontrol1005 has been down for long..Mar 4 2025, 11:16 AM

• aborrero closed subtask T387844: NodeDown Node cloudcontrol1005 has been down for long. as Resolved.

• aborrero changed the task status from In Progress to Open.Mar 4 2025, 11:20 AM

dcaro closed subtask T387845: [maintain-dbusers] move to another cloudcontrol node as Resolved.Mar 4 2025, 11:33 AM

Change #1124421 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[cloud/wmcs-cookbooks@main] inventory: place cloudcontrol1005 last on the inventory

https://gerrit.wikimedia.org/r/1124421

gerritbot added a project: Patch-For-Review.Mar 4 2025, 12:45 PM

Change #1124421 merged by Arturo Borrero Gonzalez:

[cloud/wmcs-cookbooks@main] inventory: place cloudcontrol1005 last on the inventory

https://gerrit.wikimedia.org/r/1124421

• aborrero mentioned this in rCCKBe5b22f477f2d: inventory: place cloudcontrol1005 last on the inventory.Mar 4 2025, 12:52 PM

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2025, 1:31 PM

Jclark-ctr moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Mar 5 2025, 3:52 PM

Looks like the SFP failed. Swapped it out and it looks like it's communicating again. Please check?

Confirmed that this unit came back online

fnegri mentioned this in T387946: SystemdUnitDown .Mar 6 2025, 2:25 PM

fnegri added a parent task: T387946: SystemdUnitDown .

	F58592794: image.png
	Mar 4 2025, 9:46 AM

openstack galera no recent writes 2025-03-04, suspected network hardware problemClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

openstack galera no recent writes 2025-03-04, suspected network hardware problem
Closed, ResolvedPublic
Actions

Related Objects
Search...