Page MenuHomePhabricator

fifo-log-tailer: gracefully handle missing unix socket
Closed, ResolvedPublic

Description

As found in T256449, atslog-backend spams the message Unable to read from socket to syslog if the relevant fifo-log-demux service hasn't started yet. We do already have a task for the purpose of explicitly specifying the dependency between atslog-backend/tls and their fifo-log-demux counterparts (T256467).

That being said, the spamming itself is caused by fifo-log-tailer retrying indefinitely and immediately to connect to the specified and non-existing unix socket:

root@cp4028:~# fifo-log-tailer -socket this-does-not-exist-at-all.socket
2020/11/27 10:46:22 Unable to read from socket: dial unix this-does-not-exist-at-all.socket: connect: no such file or directory
2020/11/27 10:46:22 Unable to read from socket: dial unix this-does-not-exist-at-all.socket: connect: no such file or directory
2020/11/27 10:46:22 Unable to read from socket: dial unix this-does-not-exist-at-all.socket: connect: no such file or directory
2020/11/27 10:46:22 Unable to read from socket: dial unix this-does-not-exist-at-all.socket: connect: no such file or directory
[...]

We should make sure that fifo-log-demux retries connecting less aggressively, eventually giving up after a while.

Event Timeline

ema created this task.Nov 27 2020, 10:50 AM
Restricted Application added a project: SRE. · View Herald TranscriptNov 27 2020, 10:50 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Low priority.Nov 27 2020, 10:50 AM
ema moved this task from Triage to Bug Reports on the Traffic board.

Change 644191 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/fifo-log-demux@master] fifo-log-tailer: gracefully handle missing unix socket

https://gerrit.wikimedia.org/r/644191

Change 644191 merged by Ema:
[operations/software/fifo-log-demux@master] fifo-log-tailer: gracefully handle missing unix socket

https://gerrit.wikimedia.org/r/644191

Mentioned in SAL (#wikimedia-operations) [2020-11-30T11:28:06Z] <ema> upload fifo-log-demux 0.6.2 to buster-wikimedia T268883

Mentioned in SAL (#wikimedia-operations) [2020-11-30T11:38:03Z] <ema> A:cp upgrade fifo-log-demux to 0.6.2 T268883

ema closed this task as Resolved.Nov 30 2020, 4:43 PM
ema claimed this task.

Done!

root@cp4028:~# fifo-log-tailer -socket this-does-not-exist-at-all.socket
2020/11/30 16:42:38 Unable to read from socket: dial unix this-does-not-exist-at-all.socket: connect: no such file or directory
2020/11/30 16:42:39 Unable to read from socket: dial unix this-does-not-exist-at-all.socket: connect: no such file or directory
[...]
2020/11/30 16:42:48 Could not connect to this-does-not-exist-at-all.socket after 10 attempts. Exiting.