Page MenuHomePhabricator

Flag-day change (cached values incompatible) in Gadgets extension brought translatewiki.net down
Closed, ResolvedPublic

Description

PHP Fatal error:  GadgetRepo::getStructuredList(): The script tried to execute a method or access a property of an incomplete object. Please ensure that the class definition "MediaWiki\Extension\Gadgets\Gadget" of the object you are trying to operate on was loaded _before_ unserialize() gets called or provide an autoloader to load the class definition in GadgetHooks.php:66 HookContainer.php:338 HookContainer.php:137 HookRunner.php:4103 DefaultOptionsLookup.php:108 UserOptionsManager.php:595 UserOptionsManager.php:497 UserOptionsManager.php:147 User.php:2357 RequestContext.php:384 Message.php:808 RequestContext.php:476 ContextSource.php:198 OutputPage.php:1010 Article.php:464 ViewAction.php:80 MediaWiki.php:544 MediaWiki.php:321 MediaWiki.php:910 MediaWiki.php:564 index.php:53 index.php:46 /srv/mediawiki/tags/2022-03-08_08:40:53/extensions/Gadgets/includes/GadgetRepo.php on line 94

This happened during translatewiki.net deployment window and caused 6 minute downtime for the site. I guess this is related to T303194: Gadgets extensions should not cache serialized PHP objects. During canary testing we have two code bases using same cache. WMF production only has very limited duration when this happens during deployments. But I think this would be an issue if rollback of the train is needed.

I think this could have abandoned by bumping the cache key(s).

I am not sure if anything can be done about this particular case anymore, but I would like to see someone take the ball on figuring how to reduce the likelihood of this happening again in the future.

Event Timeline

Nikerabbit renamed this task from Flag-day change in Gadgets extension brought translatewiki.net down to Flag-day change (cached values incompatible) in Gadgets extension brought translatewiki.net down.Mar 9 2022, 1:35 PM
Nikerabbit moved this task from Backlog to Mitigated on the affects-translatewiki.net board.

We had the same error (I think it was the same one) on deployment-prep, but https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Gadgets/+/768274 fixed it.

That only fixes it for one direction (forward: old data works on new code), not the other one (backward: new data works on old code).

Change 769433 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/Gadgets@wmf/1.38.0-wmf.24] wmf.24 HACK: Add forward class alias for Gadget

https://gerrit.wikimedia.org/r/769433

That only fixes it for one direction (forward: old data works on new code), not the other one (backward: new data works on old code).

Sorry, misread the task description. Adding a forward alias should reduces the risk of an outage.

Change 769434 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/Gadgets@master] Bump the cache version of Gadget

https://gerrit.wikimedia.org/r/769434

Let's also bump the cache version in order to prevent outages on other third-party sites.

Change 769433 merged by jenkins-bot:

[mediawiki/extensions/Gadgets@wmf/1.38.0-wmf.24] wmf.24 HACK: Add forward class alias for Gadget

https://gerrit.wikimedia.org/r/769433

Mentioned in SAL (#wikimedia-operations) [2022-03-09T14:57:21Z] <taavi@deploy1002> Synchronized php-1.38.0-wmf.24/extensions/Gadgets/includes: Backport: [[gerrit:769433|wmf.24 HACK: Add forward class alias for Gadget (T303391)]] (1/2) (duration: 00m 50s)

Mentioned in SAL (#wikimedia-operations) [2022-03-09T14:58:26Z] <taavi@deploy1002> Synchronized php-1.38.0-wmf.24/extensions/Gadgets/extension.json: Backport: [[gerrit:769433|wmf.24 HACK: Add forward class alias for Gadget (T303391)]] (2/2) (duration: 00m 49s)

I think this could have abandoned by bumping the cache key(s).

FWIW, Timo also filed T303194: Gadgets extensions should not cache serialized PHP objects.

Which is already mentioned in the description. Sigh. Need more coffee.

Change 769434 merged by jenkins-bot:

[mediawiki/extensions/Gadgets@master] Bump the cache version of Gadget

https://gerrit.wikimedia.org/r/769434

Zabe claimed this task.

Change 769304 had a related patch set uploaded (by Zabe; author: Zabe):

[mediawiki/extensions/Gadgets@wmf/1.38.0-wmf.25] Bump the cache version of Gadget

https://gerrit.wikimedia.org/r/769304

Change 769304 merged by jenkins-bot:

[mediawiki/extensions/Gadgets@wmf/1.38.0-wmf.25] Bump the cache version of Gadget

https://gerrit.wikimedia.org/r/769304