Page MenuHomePhabricator

Out of memory error reaching 700MBytes since 1.36.0-wmf.20
Closed, DeclinedPublicPRODUCTION ERROR

Description

https://logstash.wikimedia.org/goto/e97f32b8c6dd2fc697f1455da7e896b1

Since 1.36.0-wmf.20 got pushed to group0/group1, there are some out of memory errors showing up:

[{exception_id}] {exception_url} PHP Fatal Error from line 186 of /srv/mediawiki/php-1.36.0-wmf.20/includes/libs/rdbms/database/DatabaseMysqli.php:
Allowed memory size of 698351616 bytes exhausted (tried to allocate 4299392 bytes)
	38
[{exception_id}] {exception_url} PHP Fatal Error from line 48 of /srv/mediawiki/php-1.36.0-wmf.20/vendor/wikimedia/cdb/src/Reader/DBA.php:
Allowed memory size of 698351616 bytes exhausted (tried to allocate 1183744 bytes)
	8
[{exception_id}] {exception_url} PHP Fatal Error from line 48 of /srv/mediawiki/php-1.36.0-wmf.20/vendor/wikimedia/cdb/src/Reader/DBA.php:
Allowed memory size of 698351616 bytes exhausted (tried to allocate 1179648 bytes)
	8

Event Timeline

hashar created this task.Dec 2 2020, 6:38 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 2 2020, 6:38 PM
RhinosF1 triaged this task as Unbreak Now! priority.Dec 2 2020, 6:39 PM
RhinosF1 added a subscriber: RhinosF1.

Train blockers = UBN!

Krinkle closed this task as Declined.Dec 2 2020, 7:25 PM
Krinkle added a subscriber: Krinkle.

This was a false negative. It showed up on mediawiki-new-errors in Logstash because the "Allowed memory size of" exclusion filter on exception.message stopped working in this weeks' deployment. Hence it appeared made it seem as if OOMs were new on wmf.20 with no hits on wmf.18.

In actuality, what happened is that Monolog reporting of PHP fatal errors broke and are no longer reporting the exception field to Logstash (no message, no trace, etc.).

That issue now tracked at: T269286: Monolog update removes exception object from logstash

We checked OOMs anyway during the triage meeting and their volume has not changed week over week. Declining this task as such.