Page MenuHomePhabricator

ATS skipping certain logs due to lack of buffer space
Open, MediumPublic

Description

The following error sometimes shows up in trafficserver.service logs:

Oct 17 05:02:40 cp5001 traffic_manager[35357]: [Sep 27 05:20:26.610] {0x2b2d2d8ec700} NOTE: Skipping the current log entry for notpurge.pipe because its size (16320) exceeds the maximum payload space in a log buffer
Oct 17 05:02:40 cp5001 traffic_manager[35357]: [Oct 11 23:53:46.808] {0x2b2d2d8ec700} NOTE: Skipping the current log entry for notpurge.pipe because its size (11480) exceeds the maximum payload space in a log buffer
Oct 22 07:13:14 cp5001 traffic_manager[36320]: [Oct 17 14:42:35.338] {0x2ae8623be700} NOTE: Skipping the current log entry for notpurge.pipe because its size (8920) exceeds the maximum payload space in a log buffer

After a discussion with upstream it became clear that raising the following options might help. They both default to 9216 bytes.

  • proxy.config.log.log_buffer_size
  • proxy.config.log.max_line_size

Note that both options are undocumented, we should write the docs to be merged upstream too.

Details

Related Gerrit Patches:

Event Timeline

ema created this task.Nov 7 2019, 8:40 AM
Restricted Application added a project: Operations. · View Herald TranscriptNov 7 2019, 8:40 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.Nov 7 2019, 8:40 AM

Change 548258 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: double log_buffer_size and max_line_size

https://gerrit.wikimedia.org/r/548258

ema moved this task from Triage to Caching on the Traffic board.Nov 8 2019, 11:23 AM

Change 548258 merged by Ema:
[operations/puppet@production] ATS: double log_buffer_size and max_line_size

https://gerrit.wikimedia.org/r/548258

Change 550825 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] ATS: further increase log_buffer_size and max_line_size

https://gerrit.wikimedia.org/r/550825

Change 550825 merged by Ema:
[operations/puppet@production] ATS: further increase log_buffer_size and max_line_size

https://gerrit.wikimedia.org/r/550825

Mentioned in SAL (#wikimedia-operations) [2019-11-18T13:10:54Z] <ema> cp-ats: rolling ats-{tls,backend} restart to apply log_buffer_size config changes T237608