Page MenuHomePhabricator

thumbor haproxy trying to send syslog on wrong port
Closed, ResolvedPublic0 Story Points

Description

I noticed an increase of ICMP destination unreachable on thumbor cluster
https://grafana.wikimedia.org/d/000000366/network-performances-global?panelId=20&fullscreen&orgId=1&from=1559893910035&to=1559897539217

Investigating it, they're all:
127.0.0.1 > 127.0.0.1: ICMP 127.0.0.1 udp port 514 unreachable

For this kind of traffic:

127.0.0.1.46156 > 127.0.0.1.514 Msg: Jun  7 09:19:12 haproxy[2214]: [...]

From @fgiunchedi :

haproxy should be using the local syslog unix socket not udp syslog

Event Timeline

ayounsi triaged this task as High priority.Jun 7 2019, 9:30 AM
ayounsi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2019, 9:30 AM
Gilles assigned this task to jijiki.Jun 11 2019, 10:38 AM
jijiki moved this task from Backlog/Radar to In Progress on the User-jijiki board.Jun 18 2019, 9:45 PM

Change 517755 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] haproxy: haproxy.cfg.erb

https://gerrit.wikimedia.org/r/517755

jijiki moved this task from Backlog to Doing on the Thumbor board.Jun 18 2019, 9:51 PM
Joe moved this task from Backlog to Doing on the serviceops board.Jun 21 2019, 7:36 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-25T08:50:45Z] <jijiki> Disable puppet on thumbor* - T225284

Mentioned in SAL (#wikimedia-operations) [2019-06-25T09:02:19Z] <jijiki> Disable puppet on dbproxy* - T225284

Change 517755 merged by Effie Mouzeli:
[operations/puppet@production] haproxy: Disable global logging to syslog

https://gerrit.wikimedia.org/r/517755

Mentioned in SAL (#wikimedia-operations) [2019-06-25T09:08:50Z] <jijiki> Rolling haproxy restarts on thumbor* - T225284

It would have been nice to be subscribed or notified of this.

@jcrespo I coordinated with @Marostegui for this :)

jijiki closed this task as Resolved.Jun 25 2019, 9:34 AM