Symptoms
- issues collecting interface data through gNMI or SNMP
- TVP_PROC_STAT_FILE_OPEN_ERR: Error opening /proc/stat file in the logs
- RANCID intermittently showing +# error: timeout waiting for response from localre instead of the show version (and others) command output
- Presence of core dumps for agentd
- show interfaces hangs
Root cause
From JTAC case 2026-0728-823648 the issue is tracked in PR1957004:
The fxpc process on a host os does not properly close file descriptors, leading to file descriptor exhaustion when the system is under ddos pressure. Once the maximum limit is reached, TVP_PROC_STAT_FILE_OPEN_ERR: Error opening /proc/stat file is seen repeatedly, causing a spike in CPU usage.
This file descriptor is tied to BRCM_PKT file logging.
Fix
- What's the target release for the fix?
junos:24.2R2-S6 junos:24.2R2-S7 junos:25.2R2-S2 junos:25.4R2 junos:25.4R2-S1 junos:26.2R2 junos:26.3R1 junos:26.4R1
Firstly 25.2R2-S2 will be out approximately on 3rd week of October.
Yes, the fix is in 23.4R2-S8 and S9.
S8 is already released.
Workaround
- How to "Disable logging and file logging for BRCM_PKT to prevent further descriptor usage." ?
start shell vty fpc0 show ukern_trace handles get the ID of the BRCM_PKT trace handle set ukern_trace <# of BRCM_PKT> file-logging disable set ukern_trace <# of BRCM_PKT> logging disable
- Is there a workaround that doesn't require a device reboot ? As that can be user impacting.
No, the above disables the BRCM_PKT file logging, but the file descriptors already open for each DDOS prior will not be cleared without a reboot.
Impacted switches
Looking at Logstash, the impacted switches (showing TVP_PROC_STAT_FILE_OPEN_ERR) are:
- asw1-by27-esams (since a while)
- asw1-bw27-esams
- lsw1-a2-codfw (started recently, but also have an uptime of only 15 weeks)
- asw1-b12-drmrs
- asw1-b13-drmrs
- asw1-b4-magru
Next steps
- Reboot the "impacted switches"
- Identify on which switches the workaround needs to be applied
- Apply the workaround
- asw1-b3-magru
- asw1-b4-magru
- asw1-by27-esams
- asw1-bw27-esams
- asw1-b12-drmrs
- asw1-b13-drmrs
- lsw1-a2-codfw
- lsw1-a[3-8]-codfw
- lsw1-b[2-8]-codfw
- Discuss the need to upgrade the switches