Investigate increase syslog volume on civi1001 after buster upgrade
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Dwisehaupt
	Aug 9 2020, 4:20 PM

Description

We are seeing vastly increased syslog volume on civi1001 after the buster upgrade. This lead to increased disk use and risk of running out of space on the / partition.

It's possible that the updates shifted some logging to be in debug or more verbose setting. Investigate and determine if we need to adjust logging sessions at the application/process level or if we need to tune syslog.

Current high volume log entries include:

SmashPig-ConsumePendingQueue
drupal DonationInterface
AntifraudQueueConsumer

Note this is may be the same issue as {T269311}.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T260000 Investigate increase syslog volume on civi1001 after buster upgrade
		Open		None	T260381 Remove print_r statements that are dumping array contents to syslog

Event Timeline

Dwisehaupt created this task.Aug 9 2020, 4:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 9 2020, 4:20 PM

• DStrine moved this task from Triage to Sprint +1 on the Fundraising-Backlog board.Aug 10 2020, 7:38 PM

Verified that the new debug like logs started ~Aug 7 05:27:41 for the drupal: entries and ~Aug 7 05:32:03 for the php: entries. This coincides with turning services back up after the buster upgrade.

15:52 < ejegg> dwisehaupt: ok, looks like the code line to print_r the message has been there for a 
               while - it's logging it at WATCHDOG_INFO level, which I would have assumed we've been 
               logging at all along
15:52 < ejegg> it SHOULD go away
15:52 < ejegg> just not sure why it would have turned up now after not logging before
15:55 < dwisehaupt> ejegg: cool. i'm not sure why either.

Dwisehaupt moved this task from Triage to Up Next on the fundraising-tech-ops board.Aug 11 2020, 4:57 PM

We could filter these lines out but would need to do so in a variety of locations. It's best if we can just not print them at the source instead of doing post filtering and using additional resources. Opened T260381 for that.

• DStrine moved this task from Sprint +1 to Sprint +2 on the Fundraising-Backlog board.Aug 31 2020, 10:05 PM

Setting priority to "High" because debug logging like this can become a performance issue during peak FR season.

• DStrine moved this task from Sprint +2 to Q3 2021-2022 on the Fundraising-Backlog board.Jan 3 2021, 8:23 PM

Jgreen updated the task description. (Show Details)Feb 23 2021, 2:05 PM

Closing this task because it's ~2 years old with no updates. We're still losing a small amount of logs at peak due to excessive debug logging, but we've mitigated that somewhat by tuning rsyslog.

Jgreen changed the task status from Declined to Resolved.Jun 23 2022, 5:50 PM

Jgreen moved this task from Watching to Done on the fundraising-tech-ops board.

Investigate increase syslog volume on civi1001 after buster upgradeClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Investigate increase syslog volume on civi1001 after buster upgrade
Closed, ResolvedPublic
Actions

Related Objects
Search...