Page MenuHomePhabricator
Create Task
Maniphest T204225

ATS: log inspection at runtime
Closed, ResolvedPublic
Actions

Description

The ATS test cluster uses a named pipe as the configured log destination.

One of our requirements is to be able to read live log entries as they appear at runtime with multiple programs, both daemons and interactive ones.

An option to achieve the objective is to have a daemon that continuously reads from the named pipe and accepts connections from multiple clients. The clients can specify an optional filter and are able to see all matching logs.

@fgiunchedi mentioned nginx-log-peeker, which lacks the log filtering component but seems interesting.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• emaT207048 ATS production-ready as a backend cache layer
 ResolvedNoneT204225 ATS: log inspection at runtime

Event Timeline

• ema created this task.Sep 13 2018, 12:15 PM
Restricted Application added a project: SRE. · View Herald TranscriptSep 13 2018, 12:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• ema triaged this task as Medium priority.Sep 13 2018, 12:15 PM
• ema moved this task from Backlog to Caching on the Traffic board.
• ema added a parent task: T207048: ATS production-ready as a backend cache layer.Oct 22 2018, 9:24 AM
• ema mentioned this in T207048: ATS production-ready as a backend cache layer.Nov 8 2018, 8:16 AM
gerritbot subscribed.Nov 13 2018, 3:32 PM

Change 473232 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: logging.yaml support

https://gerrit.wikimedia.org/r/473232

gerritbot added a project: Patch-For-Review.Nov 13 2018, 3:32 PM
gerritbot added a comment.Nov 13 2018, 4:13 PM

Change 473232 merged by Ema:
[operations/puppet@production] ATS: logging.yaml support

https://gerrit.wikimedia.org/r/473232

gerritbot added a comment.Nov 14 2018, 10:19 AM

Change 473432 had a related patch set uploaded (by Ema; owner: Ema):
[operations/software/fifo-log-demux@master] fifo-log-demux 0.1

https://gerrit.wikimedia.org/r/473432

gerritbot added a comment.Nov 14 2018, 2:11 PM

Change 473503 had a related patch set uploaded (by Ema; owner: Ema):
[integration/config@master] Test fifo-log-demux with debian-glue-non-voting

https://gerrit.wikimedia.org/r/473503

gerritbot added a comment.Nov 14 2018, 4:18 PM

Change 473554 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Add module to install and configure fifo-log-demux

https://gerrit.wikimedia.org/r/473554

gerritbot added a comment.Nov 14 2018, 4:18 PM

Change 473555 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] trafficserver: configure fifo-log-demux

https://gerrit.wikimedia.org/r/473555

gerritbot added a comment.Nov 15 2018, 8:50 AM

Change 473432 merged by Ema:
[operations/software/fifo-log-demux@master] fifo-log-demux 0.1

https://gerrit.wikimedia.org/r/473432

Stashbot subscribed.Nov 15 2018, 8:58 AM

Mentioned in SAL (#wikimedia-operations) [2018-11-15T08:58:54Z] <ema> upload fifo-log-demux 0.1 to stretch-wikimedia T204225

gerritbot added a comment.Nov 15 2018, 9:16 AM

Change 473554 merged by Ema:
[operations/puppet@production] Add module to install and configure fifo-log-demux

https://gerrit.wikimedia.org/r/473554

gerritbot added a comment.Nov 15 2018, 9:23 AM

Change 473555 merged by Ema:
[operations/puppet@production] trafficserver: configure fifo-log-demux

https://gerrit.wikimedia.org/r/473555

gerritbot added a comment.Nov 15 2018, 9:48 AM

Change 473705 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: add atslog

https://gerrit.wikimedia.org/r/473705

gerritbot added a comment.Nov 15 2018, 9:54 AM

Change 473705 merged by Ema:
[operations/puppet@production] ATS: add atslog

https://gerrit.wikimedia.org/r/473705

• ema closed this task as Resolved.Nov 15 2018, 10:07 AM
gerritbot added a comment.Nov 15 2018, 10:15 AM

Change 473503 merged by jenkins-bot:
[integration/config@master] Test fifo-log-demux with debian-glue-non-voting

https://gerrit.wikimedia.org/r/473503

gerritbot added a comment.Nov 15 2018, 10:23 AM

Change 473712 had a related patch set uploaded (by Ema; owner: Ema):
[integration/config@master] Test fifo-log-demux with debian-glue

https://gerrit.wikimedia.org/r/473712

gerritbot added a comment.Nov 15 2018, 10:32 AM

Change 473712 merged by jenkins-bot:
[integration/config@master] Test fifo-log-demux with debian-glue

https://gerrit.wikimedia.org/r/473712

gerritbot added a comment.Nov 16 2018, 10:48 AM

Change 474146 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/trafficserver@master] Drop 0011-logging-broken-pipe-no-spam.patch

https://gerrit.wikimedia.org/r/474146

gerritbot added a comment.Nov 16 2018, 11:22 AM

Change 474156 had a related patch set uploaded (by Ema; owner: Ema):
[operations/debs/trafficserver@master] trafficserver (8.0.0-1wm2) stretch-wikimedia; urgency=medium

https://gerrit.wikimedia.org/r/474156

gerritbot added a comment.Nov 16 2018, 11:40 AM

Change 474146 merged by Ema:
[operations/debs/trafficserver@master] Drop 0011-logging-broken-pipe-no-spam.patch

https://gerrit.wikimedia.org/r/474146

gerritbot added a comment.Nov 16 2018, 2:22 PM

Change 474156 merged by Ema:
[operations/debs/trafficserver@master] trafficserver (8.0.0-1wm2) stretch-wikimedia; urgency=medium

https://gerrit.wikimedia.org/r/474156

Stashbot added a comment.Nov 16 2018, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-16T14:39:16Z] <ema> trafficserver 8.0.0-1wm2 uploaded to stretch-wikimedia T204225 T204209

Stashbot mentioned this in T204209: Define and deploy Icinga checks for ATS backends.Nov 16 2018, 2:39 PM
Stashbot added a comment.Nov 16 2018, 2:56 PM

Mentioned in SAL (#wikimedia-operations) [2018-11-16T14:56:20Z] <ema> upgrade cp-ats to 8.0.0-1wm2 T204225 T204209

• ema reopened this task as Open.Nov 19 2018, 5:23 PM

There's a problem with fifo-log-demux reading from the pipe, reopening!

gerritbot added a comment.Nov 19 2018, 5:23 PM

Change 474749 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: disable icinga notifications

https://gerrit.wikimedia.org/r/474749

gerritbot added a comment.Nov 19 2018, 5:26 PM

Change 474749 merged by Ema:
[operations/puppet@production] ATS: disable icinga notifications

https://gerrit.wikimedia.org/r/474749

gerritbot added a comment.Nov 19 2018, 5:50 PM

Change 474753 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: actually disable notifications

https://gerrit.wikimedia.org/r/474753

gerritbot added a comment.Nov 19 2018, 6:25 PM

Change 474753 merged by Ema:
[operations/puppet@production] ATS: actually disable notifications

https://gerrit.wikimedia.org/r/474753

• ema added a comment.EditedNov 20 2018, 9:23 AM

Yesterday all ATS hosts ran out of disk space. That's due to trafficserver logging several messages like the following:

File:/var/log/trafficserver/notpurge.pipe was closed, have dropped (N) bytes.

There are a few things going wrong and causing those messages:

  1. trafficserver closes its open logpipes upon logging.yaml config reload. It should keep them open the same way as it does for regular files.
  2. traffic_server -C verify_config, which we're currently using in the check_trafficserver_verify_config Icinga check, triggers a logging.yaml configuration reload instead of simply verifying if the syntax of the various configuration files is correct.
  3. whenever trafficserver finds its logfiles (pipes included) to be closed, it logs an error message to stdout (and thus to syslog).

While (2) isn't optimal, we need to ensure that logging keeps on working fine upon configuration reload. Hence, we should fix (1) and make sure that fifo-log-demux re-opens the pipe if it's closed.

gerritbot added a comment.Nov 20 2018, 1:20 PM

Change 474909 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: avoid verify_config for now

https://gerrit.wikimedia.org/r/474909

gerritbot added a comment.Nov 20 2018, 1:24 PM

Change 474909 merged by Ema:
[operations/puppet@production] ATS: avoid verify_config for now

https://gerrit.wikimedia.org/r/474909

• ema added a comment.Nov 21 2018, 10:22 AM
In T204225#4761225, @ema wrote:
  1. trafficserver closes its open logpipes upon logging.yaml config reload

s/closes/unlinks/. Bug filed upstream: https://github.com/apache/trafficserver/issues/4635

gerritbot added a comment.Nov 22 2018, 8:08 AM

Change 475291 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: add check_trafficserver_log_fifo

https://gerrit.wikimedia.org/r/475291

gerritbot added a comment.Nov 22 2018, 8:49 AM

Change 475291 merged by Ema:
[operations/puppet@production] ATS: add check_trafficserver_log_fifo

https://gerrit.wikimedia.org/r/475291

gerritbot added a comment.Nov 22 2018, 9:03 AM

Change 475294 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: pass fifo filename to check_trafficserver_log_fifo

https://gerrit.wikimedia.org/r/475294

gerritbot added a comment.Nov 22 2018, 9:39 AM

Change 475294 merged by Ema:
[operations/puppet@production] ATS: pass fifo filename to check_trafficserver_log_fifo

https://gerrit.wikimedia.org/r/475294

gerritbot added a comment.Nov 22 2018, 12:33 PM

Change 475313 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] fifo-log-demux: restart upon writer restart

https://gerrit.wikimedia.org/r/475313

gerritbot added a comment.Nov 22 2018, 4:35 PM

Change 475313 merged by Ema:
[operations/puppet@production] fifo-log-demux: restart upon writer restart, remove cargo cult

https://gerrit.wikimedia.org/r/475313

• ema closed this task as Resolved.Nov 26 2018, 2:55 PM
gerritbot added a comment.Nov 27 2018, 2:04 PM

Change 476018 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: log X-Cache-Status and X-Client-IP

https://gerrit.wikimedia.org/r/476018

gerritbot added a comment.Nov 27 2018, 2:07 PM

Change 476018 merged by Ema:
[operations/puppet@production] ATS: log X-Cache-Status and X-Client-IP

https://gerrit.wikimedia.org/r/476018

gerritbot added a comment.Dec 7 2018, 9:50 AM

Change 478158 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: configuration settings for named pipe logging

https://gerrit.wikimedia.org/r/478158

gerritbot added a comment.Dec 7 2018, 10:47 AM

Change 478158 merged by Ema:
[operations/puppet@production] ATS: configuration settings for named pipe logging

https://gerrit.wikimedia.org/r/478158

gerritbot added a comment.Dec 14 2018, 11:03 AM

Change 479612 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: do not log varnishcheck requests

https://gerrit.wikimedia.org/r/479612

gerritbot added a comment.Dec 14 2018, 4:24 PM

Change 479612 merged by Ema:
[operations/puppet@production] ATS: do not log varnishcheck requests

https://gerrit.wikimedia.org/r/479612

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL