Page MenuHomePhabricator

thumbor haproxy trying to send syslog on wrong port
Closed, ResolvedPublic0 Estimated Story Points

Description

I noticed an increase of ICMP destination unreachable on thumbor cluster
https://grafana.wikimedia.org/d/000000366/network-performances-global?panelId=20&fullscreen&orgId=1&from=1559893910035&to=1559897539217

Investigating it, they're all:
127.0.0.1 > 127.0.0.1: ICMP 127.0.0.1 udp port 514 unreachable

For this kind of traffic:

127.0.0.1.46156 > 127.0.0.1.514 Msg: Jun  7 09:19:12 haproxy[2214]: [...]

From @fgiunchedi :

haproxy should be using the local syslog unix socket not udp syslog

Event Timeline

ayounsi triaged this task as High priority.Jun 7 2019, 9:30 AM
ayounsi created this task.

Change 517755 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] haproxy: haproxy.cfg.erb

https://gerrit.wikimedia.org/r/517755

Mentioned in SAL (#wikimedia-operations) [2019-06-25T08:50:45Z] <jijiki> Disable puppet on thumbor* - T225284

Mentioned in SAL (#wikimedia-operations) [2019-06-25T09:02:19Z] <jijiki> Disable puppet on dbproxy* - T225284

Change 517755 merged by Effie Mouzeli:
[operations/puppet@production] haproxy: Disable global logging to syslog

https://gerrit.wikimedia.org/r/517755

Mentioned in SAL (#wikimedia-operations) [2019-06-25T09:08:50Z] <jijiki> Rolling haproxy restarts on thumbor* - T225284

It would have been nice to be subscribed or notified of this.