Page MenuHomePhabricator

ErrorException from line 1159 of /srv/mediawiki/php-1.33.0-wmf.4/includes/Message.php: PHP Warning: Invalid argument supplied for foreach()
Closed, DuplicatePublicPRODUCTION ERROR

Description

Since 2018-11-26 13:56 there is lots of:

ErrorException from line 1159 of /srv/mediawiki/php-1.33.0-wmf.4/includes/Message.php: PHP Warning: Invalid argument supplied for foreach() only on enwiki:

https://logstash.wikimedia.org/goto/9e9e2feef0cad33bd45e2b70a18085d7

I don't see any deployment at the time. Other errors that seem related:

/srv/mediawiki/php-1.33.0-wmf.4/includes/Message.php: PHP Notice: Undefined variable: key
/srv/mediawiki/php-1.33.0-wmf.4/includes/Message.php: PHP Notice: Undefined variable: message

and maybe:

[{exception_id}] {exception_url} ErrorException from line 302 of /srv/mediawiki/php-1.33.0-wmf.4/includes/Message.php: PHP Notice: Unable to unserialize: [a:8:{s:9:"interface";b:1;s:8:"language";s:2:"en";s:3:"key";s:30:"wikimediashoplink-link-tooltip";s

This may be overloading Elastic, or at least generating an alert:

[2018-11-27 10:20:33] SERVICE ALERT: icinga1001;Logstash rate of ingestion percent change compared to yesterday;CRITICAL;HARD;5;523 ge 130

Event Timeline

@Anomie Sorry to add you but e23c7c4c415885f68c4 is the only thing that was deployed at the time, it makes no sense to me, but could you discard Actor refactoring as related?

@Anomie Sorry to add you

No problem.

but e23c7c4c415885f68c4 is the only thing that was deployed at the time, it makes no sense to me, but could you discard Actor refactoring as related?

That change should only have affected group 1 wikis, while enwiki is in group 2.

Also mwrepl from mwmaint1002 tells me that enwiki still has $wgActorTableSchemaMigrationStage as 3 (SCHEMA_COMPAT_OLD), not 19 (SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD).

Ok, so this is either a "hidden" deploy, or maybe some heavy query pattern that is new. However, this seems to happen even on /wiki/Special:BlankPage ? I am confused:

/wiki/Special:BlankPage
/?curid=112901
/w/index.php?title=Nickel&printable=yes
/w/index.php?search=house+2020&title=Special%3ASearch&go=Go

@Joe @MoritzMuehlenhoff Any update to application servers yesterday at the time on the summary? (also makes no sense because it affects only enwiki but not other wiki)

@hashar noticed something possibly related to this back in wmf.3 that was transient T209582: Unable to serialize r:46: Id 46 out of range.

Memcache would indeed be a possibility- affecting all servers, all requests but only on one wiki. There are references on the errors to serialization.

I am pretty sure that is related to T209582 which I have blamed on cosmic ray. See the task there for more details but in short the memcached key for the enwiki sidebar has a reference to another object. Of course when unserializing the code, the reference mostly probably does not exist OR the object ends up pointing to a variable which is not the one expected.

TLDR: MediaWiki should not serialize references.

It is just reported as a Warning, so not user-impacting (lower priority)? It may impact to some extent the monitoring (and possibly, to some extend performance) due to extra logging.

I am pretty sure that is related to T209582

I agree. When that happens Message::unserialize() gets false for $data, so none of the members get initialized properly. Then later when things try to use that broken Message object we get the other messages when fields unexpectedly have null values.

This may have stopped since 16:34:29 (although it seems to happen in batches, so not sure yet if it will reappear) Nope, still happening.

I filed T210528: PHP/HHVM serialization incompatibility in some situations when using Serializable to explain the issue rather than the symptom, so I'm closing this as a duplicate of that task.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM