Page MenuHomePhabricator

asw1-b12-drmrs stopped reporting metrics
Closed, ResolvedPublic

Description

Both SNMP and gNMI

At exactly midnight on Nov 22nd.

On SNMP it also stopped exporting the interface description (causing LibreNMS to alert on the parent task).

Could it be something similar to T400205: Inaccurate stats reported by cr2-codfw ?

Event Timeline

Restricted Application added subscribers: Sadads, Aklapper. · View Herald Transcript
LSobanski triaged this task as Medium priority.

Opened case 2026-0107-016071

Data seems fully back for gNMI (and looks like it never went away).

For SNMP it worked briefly over the new year but it's now gone again.

Screenshot From 2026-01-07 08-19-05.png (2,080×729 px, 177 KB)

Could it be something similar to T400205: Inaccurate stats reported by cr2-codfw ?

Yeah it seems quite similar, but given that seemed to be some quirk due to the specific hardware/line-card, and the hardware here is quite different, it's probably not the exact same problem.

I guess let's see what they say. Reboot would probably fix it for now but better to get to the bottom of the issue.

Mentioned in SAL (#wikimedia-operations) [2026-01-15T07:01:17Z] <XioNoX> restart snmp and MIB processes on asw1-b12-drmrs - T413181

Mentioned in SAL (#wikimedia-operations) [2026-01-16T14:28:27Z] <XioNoX> asw1-b12-drmrs> restart statistics-service - T413181

JTAC asked us to try to reboot various deamons, none of them worked. Now they asked for a full switch reboot.
I followed up saying I'd rather troubleshot the issue properly, as a reboot doesn't guaranty that the issue won't happen again.

Mentioned in SAL (#wikimedia-operations) [2026-01-19T08:47:23Z] <XioNoX> continue asw1-b12-drmrs troubleshooting - T413181

We're currently troubleshooting why we can't see troubleshooting logs. But it can maybe be the root cause for the metrics issues.

TL;DR; we should upgrade to 23.4R2-S7 (or more recent).

Something that caught my attention is the fact that the file "messages" or sylog is filled with the following logs:
?
Jan 7 05:39:05 asw1-b12-drmrs fpc0 TVP_PROC_STAT_FILE_OPEN_ERR: Error opening /proc/stat file
?
This log has been addressed and reported via internal KBs and also through confidential PRs which have not been published yet.
In short, the error "TVP_PROC_STAT_FILE_OPEN_ERR" appears in logs or terminal output and repeated failures or abnormal behavior when running resource monitoring utilities.
?
The syslog message indicates dcpfe daemon can NOT access the file of /proc/stat in the linux host for some reason.
(/proc/stat is a virtual file in Linux that provides cumulative system activity statistics, like CPU time in various states (user, system, idle, iowait), page swaps, context switches, and interrupts, all counters that reset at boot time and are used by monitoring tools (like top, htop) to calculate real-time CPU usage and overall system health. It's crucial for performance analysis, showing how long the CPU spends on different tasks since the system started.)

There could be various reasons a file can NOT be accessed but in this case it's because too many FD(File Descriptor)s are already opened by dcpfe and its limit has been reached.
Opening a file in Linux requires one available FD to be assigned for it, but the dcpfe can't do so as it has all its available FDs already assigned opening too many /var/log/BRCM_PKT.log files.
?
The issue is tied to BRCM_PKT.log, and it occurs only when there are frequent DDoS violations in the system.

I extract the following information from internal resources:

    Each time a DDoS violation is triggered, the system creates a new file descriptor, which leads to an increasing number of open FDs.
    Currently, we cannot reduce the number of open file descriptors at runtime, and we also don't have a feature flag available to disable this behavior.
    Given these constraints, the only practical workaround for now is to prevent the switch from being subjected to DDoS attacks.

?
Looking at the logs, I see that there are also multiple notifications of ddos violations:
Jan 7 05:38:22 asw1-b12-drmrs jddosd[14006]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception TTL:aggregate exceeded its allowed bandwidth at fpc 0 for 1109 times, started at 2026-01-07 05:38:22 UTC
Jan 7 05:38:22 asw1-b12-drmrs jddosd[14006]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception L3MTU-fail:aggregate exceeded its allowed bandwidth at fpc 0 for 1109 times, started at 2026-01-07 05:38:22 UTC
?
According to further/internal resources, the permanent fix would be to upgrade the Junos software to a version with the fix for this issue: for instance 23.4R2-S7
?
Regarding the PFE commands that are timing out, I found the probable cause:
?
Workaround
?
First of all, let's confirm the BRCM_PKT handle ID:

root@QFX5120-48Y:RE:0% cprod -A fpc0 -c "show ukern_trace handles" | grep -E "Name|BRCM_PKT"
?
Then, apply the commands:
root@QFX5120-48Y:RE:0% cprod -A fpc0 -c "set ukern_trace 16 file-logging disable"
root@QFX5120-48Y:RE:0% cprod -A fpc0 -c "set ukern_trace 16 logging disable"
root@QFX5120-48Y:RE:0%
root@QFX5120-48Y:RE:0% cprod -A fpc0 -c "show ukern_trace handles" | grep -E "Name|BRCM_PKT"
ID Name Level Printf Logging Size Wrap Filter-id
*16 BRCM_PKT terse Off Off 0 1 - <========== Logging Off

This workaround will NOT work (commands will timeout) if the system is NOT rebooted and the dcpfe still is on its FD max limit.
You must apply this workaround after rebooting the system if you are already seeing the syslog message.
It is because for the workaround commands to be delivered to the dcpfe, it needs to establish new socket connections to cprod but since the dcpfe can't get any more FD assigned, no more socket opened.
(Every socket open requires a FD assigned).
?