Page MenuHomePhabricator
Create Task
Maniphest T326322

Add per-output queue monitoring for Juniper network devices
Open, MediumPublic
Actions

Description

To fully monitor the affect and performance of network devices when traffic is placed into more than one Juniper forwarding-class it would help if we could visualise the bytes/packets forwarded through each queue on egress interfaces.

These stats are exposed via the JUNIPER-COS-MIB, but not currently graphed by LibreNMS

cmooney@netmon1003:~$ snmpwalk -M +/home/cmooney/juniper_mibs -m +JUNIPER-COS-MIB -v2c -c <COMM>  lsw1-e1-eqiad.mgmt.eqiad.wmnet  JUNIPER-COS-MIB::jnxCosIfqStatsTable | grep ".590"
JUNIPER-COS-MIB::jnxCosIfqQedPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedPkts.590."mcast" = Counter64: 261599
JUNIPER-COS-MIB::jnxCosIfqQedPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedPkts.590."best-effort" = Counter64: 128584368836
JUNIPER-COS-MIB::jnxCosIfqQedPkts.590."network-control" = Counter64: 563531
JUNIPER-COS-MIB::jnxCosIfqQedPktRate.519."best-effort" = Counter64: 37590
JUNIPER-COS-MIB::jnxCosIfqQedPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedPktRate.590."best-effort" = Counter64: 8367
JUNIPER-COS-MIB::jnxCosIfqQedPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedBytes.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedBytes.590."mcast" = Counter64: 22658970
JUNIPER-COS-MIB::jnxCosIfqQedBytes.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedBytes.590."best-effort" = Counter64: 31329159923394
JUNIPER-COS-MIB::jnxCosIfqQedBytes.590."network-control" = Counter64: 188612402
JUNIPER-COS-MIB::jnxCosIfqQedByteRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedByteRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedByteRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqQedByteRate.590."best-effort" = Counter64: 1979555
JUNIPER-COS-MIB::jnxCosIfqQedByteRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedPkts.590."mcast" = Counter64: 261599
JUNIPER-COS-MIB::jnxCosIfqTxedPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedPkts.590."best-effort" = Counter64: 128584368604
JUNIPER-COS-MIB::jnxCosIfqTxedPkts.590."network-control" = Counter64: 563531
JUNIPER-COS-MIB::jnxCosIfqTxedPktRate.519."best-effort" = Counter64: 37590
JUNIPER-COS-MIB::jnxCosIfqTxedPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedPktRate.590."best-effort" = Counter64: 8367
JUNIPER-COS-MIB::jnxCosIfqTxedPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedBytes.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedBytes.590."mcast" = Counter64: 22658970
JUNIPER-COS-MIB::jnxCosIfqTxedBytes.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedBytes.590."best-effort" = Counter64: 31329159566196
JUNIPER-COS-MIB::jnxCosIfqTxedBytes.590."network-control" = Counter64: 188612402
JUNIPER-COS-MIB::jnxCosIfqTxedByteRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedByteRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedByteRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTxedByteRate.590."best-effort" = Counter64: 1979555
JUNIPER-COS-MIB::jnxCosIfqTxedByteRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPkts.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPkts.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPkts.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPktRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTailDropPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPkts.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPkts.590."best-effort" = Counter64: 232
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPkts.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPktRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPkts.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPkts.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPkts.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPktRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPkts.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPkts.590."best-effort" = Counter64: 232
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPkts.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPktRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPkts.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPkts.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPkts.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPktRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPkts.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPkts.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPkts.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPkts.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPkts.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPktRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPktRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPktRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPktRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropPktRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropBytes.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropBytes.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropBytes.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropBytes.590."best-effort" = Counter64: 357198
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropBytes.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropByteRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropByteRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropByteRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropByteRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqTotalRedDropByteRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropBytes.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropBytes.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropBytes.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropBytes.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropBytes.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropByteRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropByteRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropByteRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropByteRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpNonTcpRedDropByteRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropBytes.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropBytes.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropBytes.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropBytes.590."best-effort" = Counter64: 357198
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropBytes.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropByteRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropByteRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropByteRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropByteRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqLpTcpRedDropByteRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropBytes.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropBytes.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropBytes.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropBytes.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropBytes.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropByteRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropByteRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropByteRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropByteRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpNonTcpRedDropByteRate.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropBytes.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropBytes.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropBytes.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropBytes.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropBytes.590."network-control" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropByteRate.590."fcoe" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropByteRate.590."mcast" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropByteRate.590."no-loss" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropByteRate.590."best-effort" = Counter64: 0
JUNIPER-COS-MIB::jnxCosIfqHpTcpRedDropByteRate.590."network-control" = Counter64: 0

The desired result would be a series of stacked graphs, per-interface, for instance:

  • Transmitted bytes per queue
  • Transmitted packets per queue
  • Dropped packets (of the different types) per queue

This will aid in determining that traffic is being transmitted through the correct queues, as well as troubleshoot any issues due to drops.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

cmooney triaged this task as Medium priority.Jan 5 2023, 2:40 PM
cmooney created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 5 2023, 2:40 PM
ayounsi mentioned this in Blog Post: Multi-platform network configuration.Jun 20 2023, 2:24 PM
ayounsi subscribed.Jul 27 2023, 10:56 AM

I tested gNMI on lsw1-a1-codfw as it's not in production yet. After upgrading it to Junos 22.2.

I configured gnmic with a basic:

---
targets:
  "lsw1-a1-codfw.mgmt.codfw.wmnet:8080":
    proxy: socks5://localhost:8888
    username: ayounsi

password: [redacted]
skip-verify: True
encoding: json_ietf
outputs:
  output1:
    type: prometheus
subscriptions:
  port_stats:
    paths:
      - "/interfaces/interface/state"
    stream-mode: sample
    sample-interval: 30s
    encoding: proto

Then started a Socks proxy tunnel with ssh -D 8888 cumin1001.eqiad.wmnet
Then finally: gnmic -d subscribe --name port_stats

This exposed plenty of data to the Prometheus endpoint, including potential queues stats:
$ curl localhost:9804/metrics | grep "et-0/0/54"

interfaces_interface_state_counters_carrier_transitions{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_broadcast_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_discards{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_errors{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_fcs_errors{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_multicast_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_octets{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_pause_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_unicast_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_in_unknown_proto_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_broadcast_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_discards{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_errors{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_multicast_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_octets{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_pause_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_allocated_buffer_size{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_avg_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_bytes{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_cur_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_peak_buffer_occupancy{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_pkts{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_red_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_bytes{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_rl_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="0",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="1",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="2",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="3",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="4",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="5",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="6",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="7",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="8",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_queue_tail_drop_pkts{interface_name="et-0/0/54",out_queue_queue_number="9",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_unicast_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_counters_out_unknown_proto_pkts{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0
interfaces_interface_state_high_speed{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 100000
interfaces_interface_state_ifindex{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 523
interfaces_interface_state_last_change{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 1.690473023232174e+18
interfaces_interface_state_mtu{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 1514
interfaces_interface_state_physical_channel{interface_name="et-0/0/54",source="lsw1-a1-codfw.mgmt.codfw.wmnet:8080",subscription_name="port_stats"} 0

Even though there is a lot of work to be done if we want to go in that direction (configure gnmi on the Juniper devices, finetune gnmic, puppetize it, scrape the data, etc.) this shows that the data is there and can be exposed to Prometheus.

I couldn't find the queues stats on SONiC's gNMI's API.

cmooney added a comment.Jul 27 2023, 12:14 PM

Nice work! Bit of work to get it productionized for sure but great to see it working!

ayounsi added a comment.EditedAug 11 2023, 4:25 PM

Next steps here:

ayounsi renamed this task from Add per-output queue graphing for Juniper network devices in LibreNMS to Add per-output queue monitoring for Juniper network devices.Aug 11 2023, 4:26 PM
ayounsi mentioned this in T335027: Prometheus: ingest SONiC metrics.Aug 11 2023, 4:37 PM
gerritbot added a comment.Aug 14 2023, 7:30 AM

Change 948113 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Revert "mgmt: allow prometheus"

https://gerrit.wikimedia.org/r/948113

gerritbot added a project: Patch-For-Review.Aug 14 2023, 7:30 AM
fgiunchedi subscribed.Aug 14 2023, 8:35 AM
In T326322#9087109, @ayounsi wrote:

Next steps here:

  • Decide which hosts will run gnmic, I can think of 4 options:
    • netflowXXXX (my preferred option, as already monitoring focused and in each DC/pop, and not too much used)
    • New dedicated VMs in each DC
    • netmon hosts (in core DCs only)
    • prometheus hosts

Of these options I like the netflow one too, and in general and solution that involves deploying gnmic to each dc/pop (i.e. closest to the devices)

With my o11y hat on, this all looks sensible to me. re: TLS certificate expiration check, those come standard with http(s) checks nowadays (prometheus::blackbox::check::http). In other words adding the check above to the same place we already list/define devices in puppet will do the trick (and/or via netbox -> puppet integration? TBD)

gerritbot added a comment.Aug 14 2023, 9:19 AM

Change 948535 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Allow gNMI from netflow hosts and to Juniper devices

https://gerrit.wikimedia.org/r/948535

gerritbot added a comment.Aug 14 2023, 9:34 AM

Change 948538 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Add password support for users

https://gerrit.wikimedia.org/r/948538

gerritbot added a comment.Aug 14 2023, 9:48 AM

Change 948540 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] Enable gNMI on all devices

https://gerrit.wikimedia.org/r/948540

gerritbot added a comment.Aug 14 2023, 12:15 PM

Change 948553 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] WIP: add gNMI (+cert) check for network devices

https://gerrit.wikimedia.org/r/948553

gerritbot added a comment.Aug 25 2023, 10:26 AM

Change 952325 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Add gNMI based telemetry collection using gNMIc

https://gerrit.wikimedia.org/r/952325

gerritbot added a comment.Aug 25 2023, 12:53 PM

Change 952398 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[labs/private@master] Add mock secret for the gnmi telemetry user

https://gerrit.wikimedia.org/r/952398

gerritbot added a comment.Aug 25 2023, 12:55 PM

Change 952398 merged by Ayounsi:

[labs/private@master] Add mock secret for the gnmi telemetry user

https://gerrit.wikimedia.org/r/952398

ayounsi mentioned this in rLPRI596d888cc547: Add mock secret for the gnmi telemetry user.Aug 25 2023, 12:55 PM
gerritbot added a comment.Aug 28 2023, 5:59 AM

Change 948538 merged by jenkins-bot:

[operations/homer/public@master] Add password support for users

https://gerrit.wikimedia.org/r/948538

gerritbot added a comment.Aug 28 2023, 11:38 AM

Change 948535 merged by jenkins-bot:

[operations/homer/public@master] Allow gNMI from netflow hosts and to Juniper devices

https://gerrit.wikimedia.org/r/948535

gerritbot added a comment.Aug 30 2023, 9:51 AM

Change 948540 merged by jenkins-bot:

[operations/homer/public@master] Enable gNMI on access and cloud switches

https://gerrit.wikimedia.org/r/948540

ayounsi added a comment.Aug 30 2023, 10:08 AM

@cmooney I came across https://www.juniper.net/documentation/us/en/software/junos/interfaces-telemetry/topics/ref/statement/class-of-service-schedulers-queue-monitoring.html not sure exactly what it does though but seems relevant :)

gerritbot added a comment.Aug 30 2023, 12:14 PM

Change 952325 merged by Ayounsi:

[operations/puppet@production] Add gNMI based telemetry collection using gNMIc

https://gerrit.wikimedia.org/r/952325

ayounsi added a comment.Aug 30 2023, 1:38 PM

I rolled the certificate to all the cloudsw, cr, and asw devices.
I enabled gnmic on all the cloudsw and asw devices.
I configured gnmic to pull the data from all the asw devices.

So far the issues:

  1. The virtual chassis devices (maybe not related to the VC but to the device platform) are not returning anything on their grpc endpoint (not even tls cert)
  2. L3 switches have various combination of FQDN configured on their TLS cert vs. FQDN used in gnmic to pull data from them vs. if the grpc service is configured to run in the mgmt_junos VRF

I'll investigate the first point and streamline the 2nd.

In the meantime it works fine for the remaining devices (mostly eqiad E/F):
prometheus1005:~$ curl netflow1002:9804/metrics

gerritbot added a comment.Aug 30 2023, 1:46 PM

Change 953632 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] gnmi: allow v6 connectivity

https://gerrit.wikimedia.org/r/953632

gerritbot added a comment.Aug 30 2023, 1:55 PM

Change 953636 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] gnmi: remove mgmt_junos restriction

https://gerrit.wikimedia.org/r/953636

gerritbot added a comment.Aug 30 2023, 1:57 PM

Change 953632 merged by jenkins-bot:

[operations/homer/public@master] gnmi: allow v6 connectivity

https://gerrit.wikimedia.org/r/953632

gerritbot added a comment.Aug 30 2023, 1:57 PM

Change 953636 merged by jenkins-bot:

[operations/homer/public@master] gnmi: remove mgmt_junos restriction

https://gerrit.wikimedia.org/r/953636

gerritbot added a comment.Aug 31 2023, 8:12 AM

Change 953961 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/homer/public@master] gNMI: remove from cloudsw, add to cr

https://gerrit.wikimedia.org/r/953961

gerritbot added a comment.Aug 31 2023, 8:15 AM

Change 953961 merged by jenkins-bot:

[operations/homer/public@master] gNMI: remove from cloudsw, add to cr

https://gerrit.wikimedia.org/r/953961

gerritbot added a comment.Aug 31 2023, 8:42 AM

Change 953964 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] gNMI: collect data from core routers

https://gerrit.wikimedia.org/r/953964

gerritbot added a comment.Aug 31 2023, 9:26 AM

Change 953969 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] Prometheus: scrape gNMIc endpoint

https://gerrit.wikimedia.org/r/953969

gerritbot added a comment.Aug 31 2023, 9:30 AM

Change 953964 merged by Ayounsi:

[operations/puppet@production] gNMI: collect data from core routers

https://gerrit.wikimedia.org/r/953964

gerritbot added a comment.Aug 31 2023, 9:41 AM

Change 953969 merged by Ayounsi:

[operations/puppet@production] Prometheus: scrape gNMIc endpoints

https://gerrit.wikimedia.org/r/953969

gerritbot added a comment.Aug 31 2023, 1:14 PM

Change 954053 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: fix gnmi relabel

https://gerrit.wikimedia.org/r/954053

gerritbot added a comment.Aug 31 2023, 1:30 PM

Change 954053 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: fix gnmi relabel

https://gerrit.wikimedia.org/r/954053

gerritbot added a comment.Aug 31 2023, 1:37 PM

Change 954055 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: drop 'cluster' for gnmi job

https://gerrit.wikimedia.org/r/954055

gerritbot added a comment.Aug 31 2023, 1:40 PM

Change 954055 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: drop 'cluster' for gnmi job

https://gerrit.wikimedia.org/r/954055

ayounsi added a subscriber: MoritzMuehlenhoff.EditedAug 31 2023, 1:55 PM

We have data https://grafana.wikimedia.org/d/iUATvNzSz/network-queues !
And a doc: https://wikitech.wikimedia.org/wiki/Network_telemetry

The last few things remaining are:

gerritbot added a comment.Aug 31 2023, 2:12 PM

Change 954066 had a related patch set uploaded (by Ayounsi; author: Ayounsi):

[operations/puppet@production] gNMIc: add interface description as metrics tag

https://gerrit.wikimedia.org/r/954066

gerritbot added a comment.Aug 31 2023, 2:25 PM

Change 954066 merged by Ayounsi:

[operations/puppet@production] gNMIc: add interface description as metrics tag

https://gerrit.wikimedia.org/r/954066

ayounsi mentioned this in T347461: Build and package gnmic.Sep 27 2023, 8:34 AM
ayounsi added a subtask: T347461: Build and package gnmic.
ayounsi added a comment.EditedSep 27 2023, 1:29 PM

To keep it somewhere for later, on Dell SONiC it should be on the /openconfig-qos:qos/interfaces path.
Grouping it by source/interface_interface-id/pfc-priority_dot1p and displaying it by "events" gNMI returns this:

{
  "name": "interfaces-states",
  "timestamp": 1695808554638790732,
  "tags": {
    "interface_interface-id": "Ethernet9",
    "pfc-priority_dot1p": "7",
    "source": "lsw1-e8-eqiad.mgmt.eqiad.wmnet:8080",
    "subscription-name": "interfaces-states"
  },
  "values": {
    "/openconfig-qos:qos/interfaces/interface/pfc/pfc-priorities/pfc-priority/config/dot1p": 7,
    "/openconfig-qos:qos/interfaces/interface/pfc/pfc-priorities/pfc-priority/config/enable": false,
    "/openconfig-qos:qos/interfaces/interface/pfc/pfc-priorities/pfc-priority/dot1p": 7,
    "/openconfig-qos:qos/interfaces/interface/pfc/pfc-priorities/pfc-priority/state/dot1p": 7,
    "/openconfig-qos:qos/interfaces/interface/pfc/pfc-priorities/pfc-priority/state/enable": false,
    "/openconfig-qos:qos/interfaces/interface/pfc/pfc-priorities/pfc-priority/state/statistics/pause-frames-rx": 0,
    "/openconfig-qos:qos/interfaces/interface/pfc/pfc-priorities/pfc-priority/state/statistics/pause-frames-tx": 0
  }
}

Maybe when QoS is configured on the switch, more information will be displayed as pause-frames only is quite useless for us.

ayounsi added a comment.Oct 25 2023, 7:39 AM

Update on the JTAC case, they're still working on it, they tested it without SSL and it worked fine, but:

Hope you are doing well, after trying to implement SSL, the same seems to be not working, hence we are working on the next steps with higher resources by tomorrow morning. we will update you with the next steps.

cmooney added a comment.EditedMar 8 2024, 12:50 PM

@ayounsi finally got back to this for a closer look. Really great work, I tried to make a device-centric dashboard here also.

Couple of other thoughts:

gnmic interval

One thing I notice that perhaps we should change is the "sample-interval" in the gnmic config. I notice we have it set to 30 seconds? But our prometheus scraper is set to only run every 60 seconds so I'm not sure it makes sense to be more frequent on the gnmic side. Both using the same interval is probably best, and 60 seconds sounds reasonable as an interval (big improvement on our 5 mins in LibreNMS, you can see how much higher the peaks are in Grafana).

Allow all interface counters / events

Now that we know it's working, and the CPU hit on netflow boxes hasn't been huge, I wonder if we might be able to allow the entire set of data exposed in /interfaces/interface/state through to Prometheus (from your initial test output this seems to be the regular IF-MIB stats). I could see that being useful for qos graphing/alerting, but also just in general for making some Grafana dashboards of traffic?

We're already gathering it from the devices and just filtering it with gnmic, so it won't increase load on the network kit. Netflow VM CPU usage also looks ok, and while storage is a concern I think in terms of of overall Grafana it's not a huge increase?

As to how to do it I am not 100% sure about how the gnmic processors work though. The per-queue stats correctly have a grouping function like this:

group-by-queue-number:
  event-group-by:
    tags:
    - target
    - interface_name
    - out-queue_queue-number

For the regular (interface-wide) stats we still want to group by 'target' and 'interface_name', but there is no 'out-queue_queue-number' tag on those.

So I'm not 100% sure how it would behave if we remove / modify the event-allow rule to let series without that tag through. Would it just group by the available tags for the series? Or would we need to somehow create another event-group-by processor which only lists the two tags we'll have on those other series?

I guess we can maybe toy around with it seeing as it's not used for anything right now? I looked at the docs but it wasn't clear to me how it would behave.

cmooney added a comment.EditedMar 8 2024, 1:56 PM

@fgiunchedi wondering if you'd any thoughts on the above suggestion to allow more series through from the gnmic pipeline?

I can do some tests to get a better scope of the number we're talking, but effectively it's the same as SNMP IF-MIB data, or node_network prometheus stats, but for the network kit. So shouldn't be anything insane.

fgiunchedi added a comment.Mar 8 2024, 4:26 PM

Yeah having some ballpark numbers will be a great help @cmooney, unless we're talking hundreds of thousands more metrics than we have now I think we're good to go, tens of thousands we can do without much effort/resources

cmooney added a comment.Mar 8 2024, 7:36 PM
In T326322#9615636, @fgiunchedi wrote:

Yeah having some ballpark numbers will be a great help @cmooney, unless we're talking hundreds of thousands more metrics than we have now I think we're good to go, tens of thousands we can do without much effort/resources

Great. I'll do some lab tests to quantify the number we're talking about and make sure the gnmic is grouping them as well need and let you know. Thanks!

cmooney added a comment.Mar 21 2024, 4:05 PM
In T326322#9130092, @ayounsi wrote:

@cmooney I came across https://www.juniper.net/documentation/us/en/software/junos/interfaces-telemetry/topics/ref/statement/class-of-service-schedulers-queue-monitoring.html not sure exactly what it does though but seems relevant :)

Sorry I missed this one. Em yeah hard to find info on what it's doing. The stats already been allowed through give us current/average/peak buffer occupancy on a per-queue basis, but only from the core routers / MX platform. I'll look into this, perhaps it will enable the same for the QFX.

dcaro subscribed.Mar 21 2024, 4:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL