Page MenuHomePhabricator
Create Task
Maniphest T402938

KernelErrors Server cloudcephosd1052 logged kernel errors
Closed, ResolvedPublic
Actions

Description

Common information

  • alertname: KernelErrors
  • category: priority_err
  • cluster: wmcs
  • instance: cloudcephosd1052:9100
  • job: node
  • prometheus: ops
  • severity: critical
  • site: eqiad
  • source: prometheus
  • team: wmcs

Firing alerts

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedJclark-ctrT402938 KernelErrors Server cloudcephosd1052 logged kernel errors
Unknown Object (Task)

Event Timeline

phaultfinder created this task.Aug 26 2025, 3:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 26 2025, 3:49 PM
fnegri subscribed.Aug 26 2025, 3:54 PM

Some errors were logged while setting up this new host.

The lvm2 errors can be ignored (race condition, same as T402475).

The firmware errors might require more investigation.

Aug 26 14:01:51 cloudcephosd1052 kernel: ice 0000:22:00.0: firmware: failed to load intel/ice/ddp/ice-303ea7ffff2f051c.pkg (-2)
Aug 26 14:01:51 cloudcephosd1052 kernel: firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware
Aug 26 14:01:51 cloudcephosd1052 kernel: ice 0000:22:00.0: Fail adding RX LLDP rule on VSI 6 error: ICE_ERR_AQ_ERROR
Aug 26 14:01:51 cloudcephosd1052 kernel: ice 0000:22:00.1: firmware: failed to load intel/ice/ddp/ice-303ea7ffff2f051c.pkg (-2)
Aug 26 14:01:51 cloudcephosd1052 kernel: ice 0000:22:00.1: Fail adding RX LLDP rule on VSI 8 error: ICE_ERR_AQ_ERROR
Aug 26 15:39:54 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Aug 26 15:39:54 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Aug 26 15:39:55 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Aug 26 15:39:55 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Aug 26 15:39:56 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Aug 26 15:39:56 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Aug 26 15:39:56 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Aug 26 15:39:56 cloudcephosd1052 lvm2-activation-generator: lvmconfig failed
Andrew assigned this task to Jclark-ctr.Aug 26 2025, 4:00 PM
Andrew added subscribers: Jclark-ctr, Andrew.

@Jclark-ctr We're pretty sure those lvm issues can be ignored, but the firmware errors are concerning; do you know what fw version we want to run for that intel network card?

Andrew mentioned this in T395910: cloudcephosd10[48-52] service implementation.Aug 26 2025, 4:03 PM
Jclark-ctr added a comment.Aug 26 2025, 5:30 PM

Firmware was 23.0.8
Updating to 24.0.5

Jclark-ctr added a comment.Aug 26 2025, 5:37 PM

@Andrew If errors continue, we might need to upgrade to a newer version of Debian. I imaged the server with Bullseye today

Jclark-ctr added a project: ops-eqiad.Aug 26 2025, 6:25 PM
Restricted Application added a project: DC-Ops. · View Herald TranscriptAug 26 2025, 6:25 PM
Jclark-ctr moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Aug 26 2025, 6:25 PM
Maintenance_bot added a project: SRE.Aug 26 2025, 6:29 PM
Jclark-ctr added a subscriber: wiki_willy.Aug 27 2025, 2:43 PM

@wiki_willy The error Returned in Dmesg. The best option might be to purchase a 25G Broadcom NIC to avoid future problems with an oddball in the cluster.

wiki_willy added a subscriber: RobH.Aug 27 2025, 2:58 PM

++ @RobH - can you work with John on getting a 25g Broadcom NIC for this one?

In T402938#11124390, @Jclark-ctr wrote:

@wiki_willy The error Returned in Dmesg. The best option might be to purchase a 25G Broadcom NIC to avoid future problems with an oddball in the cluster.

RobH mentioned this in Unknown Object (Task).Aug 27 2025, 3:03 PM
RobH added a comment.Aug 27 2025, 3:08 PM
In T402938#11124505, @wiki_willy wrote:

++ @RobH - can you work with John on getting a 25g Broadcom NIC for this one?

In T402938#11124390, @Jclark-ctr wrote:

@wiki_willy The error Returned in Dmesg. The best option might be to purchase a 25G Broadcom NIC to avoid future problems with an oddball in the cluster.

T403074 filed and quote requested.

RobH added a subtask: Unknown Object (Task).Aug 27 2025, 6:13 PM
Jclark-ctr added a comment.Aug 28 2025, 12:33 PM

Resolving this ticket since we will be replacing Nic with one that matches existing servers

Jclark-ctr closed this task as Resolved.Aug 28 2025, 12:33 PM
Jclark-ctr closed subtask Unknown Object (Task) as Resolved.Sep 3 2025, 3:28 PM
fnegri mentioned this in T404300: Remove KernelErrors alerts.Sep 11 2025, 9:04 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits