Phabricator error log PHP entries truncated
Open, LowPublicBUG REPORT
Actions

Assigned To

None

Authored By

	Aklapper
	Mar 26 2024, 2:55 AM

Description

Go to either /var/log/apache2/phabricator_error.log on phab1004.eqiad.wmnet (or shinier https://logstash.wikimedia.org/app/dashboards#/view/AWt2XRVF0jm7pOHZjNIV ).
See log entries truncated at the beginning which makes fixing things pretty impossible.

The vast majority of entries correctly starts with Got error 'PHP message: [YYYY-MM-DD HH:MM:SS].

However a few lines are truncated and start with e.g.
Got error 'est, PhutilAggregateException) called at [<phorge>/src/...
Got error 'ionConfiguration::runHTTPRequest...
Got error 'orge>/src/infrastructure/markup/render.php...
Got error 'ex.php:35]PHP message...
You get the pattern.

log_errors_max_len seems not set in /etc/php/7.4/fpm/php.ini (as long as we're on PHP7) but some of those resulting messages are way less than the default 1024 bytes anyway, e.g. 665 bytes.
Anyone got an idea where to start investigating?

Event Timeline

Aklapper created this task.Mar 26 2024, 2:55 AM

I located some instances here. The count of these truncated logs is very small - 131 of 30,249 in 30 days, 0.4% of log volume.

I also located some logs on phab1004 matching the patterns you describe. It seems likely that these are fragments of very long stack traces. I located a stack trace that had PhutilAggregateException that had 8071 characters in its message and was clipped at the end.

Given the "Got error" prefix, this seems most likely something Phabricator or PHP7 is doing through a formatter. The logging pipeline does not add this prefix.

HTH!

Thanks, that's helpful (also to know that we do not "lose" messages but just split).
Cannot find Got error in the Phorge/Arcanist codebases so it's probably some PHP7 formatter... somewhere.

Aklapper moved this task from To Triage to Infrastructure on the Phabricator board.Apr 21 2024, 3:26 PM

Note to myself: wondering if that's either log rotation or some Apache variable, per https://stackoverflow.com/questions/11021575/how-to-make-apache-error-log-entries-longer/43281399#43281399

Because on phab1004, the definition of access.format in /etc/php/7.4/fpm/pool.d/www.conf starts with the date/time stamp

Phabricator error log PHP entries truncatedOpen, LowPublicBUG REPORTActions

Description

Event Timeline

Phabricator error log PHP entries truncated
Open, LowPublicBUG REPORT
Actions