Page MenuHomePhabricator
Create Task
Maniphest T269278

Out of memory error reaching 700MBytes since 1.36.0-wmf.20
Closed, DeclinedPublicPRODUCTION ERROR
Actions

Description

https://logstash.wikimedia.org/goto/e97f32b8c6dd2fc697f1455da7e896b1

Since 1.36.0-wmf.20 got pushed to group0/group1, there are some out of memory errors showing up:

[{exception_id}] {exception_url} PHP Fatal Error from line 186 of /srv/mediawiki/php-1.36.0-wmf.20/includes/libs/rdbms/database/DatabaseMysqli.php:
Allowed memory size of 698351616 bytes exhausted (tried to allocate 4299392 bytes)
	38
[{exception_id}] {exception_url} PHP Fatal Error from line 48 of /srv/mediawiki/php-1.36.0-wmf.20/vendor/wikimedia/cdb/src/Reader/DBA.php:
Allowed memory size of 698351616 bytes exhausted (tried to allocate 1183744 bytes)
	8
[{exception_id}] {exception_url} PHP Fatal Error from line 48 of /srv/mediawiki/php-1.36.0-wmf.20/vendor/wikimedia/cdb/src/Reader/DBA.php:
Allowed memory size of 698351616 bytes exhausted (tried to allocate 1179648 bytes)
	8

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedReleasehasharT263186 1.36.0-wmf.20 deployment blockers
 DeclinedPRODUCTION ERRORNoneT269278 Out of memory error reaching 700MBytes since 1.36.0-wmf.20

Event Timeline

hashar created this task.Dec 2 2020, 6:38 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptDec 2 2020, 6:38 PM
RhinosF1 triaged this task as Unbreak Now! priority.Dec 2 2020, 6:39 PM
RhinosF1 subscribed.

Train blockers = UBN!

Krinkle closed this task as Declined.Dec 2 2020, 7:25 PM
Krinkle subscribed.

This was a false negative. It showed up on mediawiki-new-errors in Logstash because the "Allowed memory size of" exclusion filter on exception.message stopped working in this weeks' deployment. Hence it appeared made it seem as if OOMs were new on wmf.20 with no hits on wmf.18.

In actuality, what happened is that Monolog reporting of PHP fatal errors broke and are no longer reporting the exception field to Logstash (no message, no trace, etc.).

That issue now tracked at: T269286: Monolog update removes exception object from logstash

We checked OOMs anyway during the triage meeting and their volume has not changed week over week. Declining this task as such.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL