Limited to NDA as security related.

After pushing the new firewall policies to the mgmt routers, this blocked flow started to show up in the logs by bursts:

Jul 17 13:04:55 mr1-codfw RT_FLOW: RT_FLOW_SESSION_DENY: session denied 128.0.0.16/1534->191.255.255.255/1534 None 17(0) default-deny(global) mgmt production UNKNOWN UNKNOWN N/A(N/A) ge-0/0/0.0 UNKNOWN policy deny

It doesn't seem to impact anything but shouldn't happen, it also only happen in codfw. And seems to be yet another rabbit hole. Opening this task to document my findings.

I ran a packet capture on mr1-codfw capturing that specific source IP for ~17min and captured 1392 packets.

Source IP 128.0.0.16
Source port 1534/udp
Destination IP: 191.255.255.255 (PTR: 191-255-255-255.dsl.telesp.net.br, but most likely a test IP)
Destination port 1534/udp

Here is the breakdown of source MAC addresses:

count "mac" - matching IP - PTR
    138 "00:a0:a5:79:9c:14" - 10.193.0.12 - re0.cr2-codfw.mgmt.codfw.wmnet
    138 "00:a0:a5:79:a3:42" - 10.193.0.13 - re1.cr2-codfw.mgmt.codfw.wmnet
    138 "00:a0:a5:7d:04:d2" - 10.193.0.10 - re0.cr1-codfw.mgmt.codfw.wmnet
    140 "10:0e:7e:b3:16:01" - 10.193.0.16 - asw-a-codfw.mgmt.codfw.wmnet
    140 "10:0e:7e:b3:50:c1" - 10.193.0.17 - asw-b-codfw.mgmt.codfw.wmnet
    140 "10:0e:7e:b4:6c:81" - 10.193.0.19 - asw-d-codfw.mgmt.codfw.wmnet
    140 "10:0e:7e:b5:9e:c1" - 10.193.0.18 - asw-c-codfw.mgmt.codfw.wmnet
    140 "7c:e2:ca:12:26:c2" - 10.193.0.57 - fasw-c-codfw:vme.0
    140 "b0:c6:9a:db:05:82" - 10.193.0.3 - msw1-codfw.mgmt.codfw.wmnet
    138 "ec:38:73:e8:98:48" - unknown - on msw-c5-codfw - vendor: Juniper Networks

Destination MAC is the next hop for the production network, at the exception of a few broadcast MACs.

Payload is either one of those two:

TCF2ID=TCP:128.0.0.101:1534
Name=TCF Agent
OSName=Linux 3.10.100-ovp-rt110-WR6.0.0.31_preempt-rt
UserName=rootAgent
ID=e60f1a9e-331e-496d-b6fa-0c6cde965329
TransportName=TCP
ServiceManagerID=e60f1a9e-331e-496d-b6fa-0c6cde965329-0
Port=1534
Host=128.0.0.101

or

TCF2ID=TCP:128.0.0.16:1534
Name=TCF Agent
OSName=Linux 3.10.100-ovp-rt110-WR6.0.0.31_preempt-rt
UserName=rootAgent
ID=e60f1a9e-331e-496d-b6fa-0c6cde965329
TransportName=TCP
ServiceManagerID=e60f1a9e-331e-496d-b6fa-0c6cde965329-0
Port=1534
Host=128.0.0.16

Running a packet capture on some of the mgmt interface of the devices listed above return nothing.

So my guesses are either
1/ A device is sending those packets while spoofing the source MAC
2/ The Juniper devices have a process running at a lower level than the packet capture (preferred guess)

TCF might be related to https://www.eclipse.org/tcf/

Next step is to open a ticket with JTAC.