Page MenuHomePhabricator

dropped packets to kafkamon 9000/tcp
Closed, ResolvedPublic

Description

From https://logstash.wikimedia.org/goto/fe382b6af2eddfdd241c0b9504482768

It seems like prometheus1003/4 and 2003/4 are trying to reach kafkamon1001 and 2001 on port 9000/tcp but iptables is dropping the packets.

Not sure if it's expected, but pointing it out in case it's not.

Details

Related Gerrit Patches:

Event Timeline

ayounsi triaged this task as Medium priority.Nov 20 2019, 8:42 PM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2019, 8:42 PM

In addition prometheus2003/4.codfw.wmnet are also trying to reach 9700/tcp on kafkamon2001 only.

Indeed that's prometheus@analytics trying to reach burrow-exporter on port 9000 on kafkamon hosts, burrow-exporter is listening there but clearly no ferm. @Ottomata @elukey is port 9000 a legacy configuration we need to clean up or expected to be working ?

Change 552393 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] role::prometheus::analytics: remove old burrow configuration

https://gerrit.wikimedia.org/r/552393

Change 552393 merged by Elukey:
[operations/puppet@production] role::prometheus::analytics: remove old burrow configuration

https://gerrit.wikimedia.org/r/552393

Wasn't able to run puppet on prometheus1003 since it is disable, but we should be close to fix the issue :)

fgiunchedi closed this task as Resolved.Dec 9 2019, 11:14 AM
fgiunchedi claimed this task.

Done! packet drops are gone