Page MenuHomePhabricator

Investigate increase syslog volume on civi1001 after buster upgrade
Open, HighPublic

Description

We are seeing vastly increased syslog volume on civi1001 after the buster upgrade. This lead to increased disk use and risk of running out of space on the / partition.

It's possible that the updates shifted some logging to be in debug or more verbose setting. Investigate and determine if we need to adjust logging sessions at the application/process level or if we need to tune syslog.

Current high volume log entries include:

  • SmashPig-ConsumePendingQueue
  • drupal DonationInterface
  • AntifraudQueueConsumer

Note this is may be the same issue as {T269311}.

Event Timeline

Verified that the new debug like logs started ~Aug 7 05:27:41 for the drupal: entries and ~Aug 7 05:32:03 for the php: entries. This coincides with turning services back up after the buster upgrade.

15:52 < ejegg> dwisehaupt: ok, looks like the code line to print_r the message has been there for a 
               while - it's logging it at WATCHDOG_INFO level, which I would have assumed we've been 
               logging at all along
15:52 < ejegg> it SHOULD go away
15:52 < ejegg> just not sure why it would have turned up now after not logging before
15:55 < dwisehaupt> ejegg: cool. i'm not sure why either.

We could filter these lines out but would need to do so in a variety of locations. It's best if we can just not print them at the source instead of doing post filtering and using additional resources. Opened T260381 for that.

Jgreen triaged this task as High priority.Oct 27 2020, 8:55 PM
Jgreen added a subscriber: Jgreen.

Setting priority to "High" because debug logging like this can become a performance issue during peak FR season.