Page MenuHomePhabricator

Coworker log issues
Open, Needs TriagePublic

Description

We have 3 issues with our coworker logging at the moment -

  1. my current issue - there is no enough detail hitting the syslog to determine why it is failing on staging. I have an upstream patch (now merged) to address that https://lab.civicrm.org/dev/coworker/-/merge_requests/7 (which would probably make the request to @Dwisehaupt to set staging to use the log-to-file flag redundant). However if we want to test it on staging we would need to download the phar per https://lab.civicrm.org/dev/coworker/-/blob/master/doc/develop.md from https://download.civicrm.org/coworker or /coworker-EDGE.phar

https://download.civicrm.org/coworker/coworker-EDGE-v0.5-10-gbefd164.phar - ideally we would test it there before leaning on Tim to put out a formal release

  1. when we log from CiviCRM for command line based activity we log to stdout. This causes problems with coworker. Tim suggested that we could log to stderr instead. This is an easy codechange but I need to check in with @Dwisehaupt what the impact of switching from stdout to stderr would be on our job logs & with @Ejegg / others if it would impact our practices. Alternatively we could detect coworker and log to stderr rather than stdout conditionally. I suspect logging to stderr in general is more correct but I'm not sure if it would impact us
  1. we overlog information from coworker - this might be purely as a result of 2 - ie we ignore these errors "

Jan 23 12:45:17 coworker[1448965]: #033[31m[2024-01-23T12:45:17.882285+00:00] Pipe[ctl#1].ERROR: Received unexpected response line: "{"jsonrpc":"2.0","result":[],"id":667944}"#033[0m" - Tim suggested we don't worry about this until 2 is resolved

Links

Upstream issue https://lab.civicrm.org/dev/coworker/-/issues/13
Phar publishing https://lab.civicrm.org/dev/coworker/-/blob/master/doc/develop.md

Related Objects

Event Timeline

Did some digging into process control. When entries come to process control, they are logged to output files based on the stream they come in on. Anything that comes in on stdout gets tagged as INFO in the output log and anything that comes across stderrgets tagged as ERROR. This is only for the process-control log files.

For syslog, process-control doesn't log that level of detail, only high level things such as job start, stop, log file, etc.

I'm unsure how the delineation of INFO vs ERROR is used in practice with the process-control logs. If we shift civi logging to send more information across stderr instead of stdout, it will affect what we see. It may have no effect in day to day use but is something we need to be aware of.

Relevant line in runner:
https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/process-control/+/refs/heads/master/processcontrol/runner.py#89

Which calls out to the output_streamer:
https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia/fundraising/process-control/+/refs/heads/master/processcontrol/output_streamer.py#44

Change 994392 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Add StdErr Monolog logger

https://gerrit.wikimedia.org/r/994392

Change 994392 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add StdErr Monolog logger

https://gerrit.wikimedia.org/r/994392