Page MenuHomePhabricator

Circular dependency when creating service! ContentLanguage
Closed, ResolvedPublic

Description

Not the first time I've seen this... A refresh seems to fix it, but it shouldn't be happening in the first place...

[f4214c14e71c7f57364ba051] /wiki/Special:Version RuntimeException from line 448 of /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php: Circular dependency when creating service! ContentLanguage -> ResourceLoader -> RevisionLookup -> RevisionStore -> RevisionStoreFactory -> BlobStoreFactory -> ContentLanguage

Backtrace:

#0 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#1 /var/www/wiki/mediawiki/core/includes/MediaWikiServices.php(537): Wikimedia\Services\ServiceContainer->getService(string)
#2 /var/www/wiki/mediawiki/core/includes/ServiceWiring.php(107): MediaWiki\MediaWikiServices->getContentLanguage()
#3 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#4 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#5 /var/www/wiki/mediawiki/core/includes/MediaWikiServices.php(456): Wikimedia\Services\ServiceContainer->getService(string)
#6 /var/www/wiki/mediawiki/core/includes/ServiceWiring.php(632): MediaWiki\MediaWikiServices->getBlobStoreFactory()
#7 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#8 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#9 /var/www/wiki/mediawiki/core/includes/MediaWikiServices.php(914): Wikimedia\Services\ServiceContainer->getService(string)
#10 /var/www/wiki/mediawiki/core/includes/ServiceWiring.php(625): MediaWiki\MediaWikiServices->getRevisionStoreFactory()
#11 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#12 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#13 /var/www/wiki/mediawiki/core/includes/MediaWikiServices.php(906): Wikimedia\Services\ServiceContainer->getService(string)
#14 /var/www/wiki/mediawiki/core/includes/ServiceWiring.php(611): MediaWiki\MediaWikiServices->getRevisionStore()
#15 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#16 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#17 /var/www/wiki/mediawiki/core/includes/MediaWikiServices.php(890): Wikimedia\Services\ServiceContainer->getService(string)
#18 /var/www/wiki/mediawiki/core/includes/Revision.php(76): MediaWiki\MediaWikiServices->getRevisionLookup()
#19 /var/www/wiki/mediawiki/core/includes/Revision.php(138): Revision::getRevisionLookup()
#20 /var/www/wiki/mediawiki/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(139): Revision::newFromTitle(Title)
#21 /var/www/wiki/mediawiki/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(108): MediaWikiGadgetsDefinitionRepo->fetchStructuredList()
#22 /var/www/wiki/mediawiki/core/includes/libs/objectcache/wancache/WANObjectCache.php(1426): MediaWikiGadgetsDefinitionRepo->{closure}(boolean, integer, array, NULL)
#23 /var/www/wiki/mediawiki/core/includes/libs/objectcache/wancache/WANObjectCache.php(1280): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array)
#24 /var/www/wiki/mediawiki/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(115): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#25 /var/www/wiki/mediawiki/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(31): MediaWikiGadgetsDefinitionRepo->loadGadgets()
#26 /var/www/wiki/mediawiki/extensions/Gadgets/includes/GadgetHooks.php(134): MediaWikiGadgetsDefinitionRepo->getGadgetIds()
#27 /var/www/wiki/mediawiki/core/includes/Hooks.php(174): GadgetHooks::registerModules(ResourceLoader)
#28 /var/www/wiki/mediawiki/core/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#29 /var/www/wiki/mediawiki/core/includes/ServiceWiring.php(597): Hooks::run(string, array)
#30 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#31 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#32 /var/www/wiki/mediawiki/core/includes/MediaWikiServices.php(874): Wikimedia\Services\ServiceContainer->getService(string)
#33 /var/www/wiki/mediawiki/core/includes/cache/localisation/LocalisationCache.php(1035): MediaWiki\MediaWikiServices->getResourceLoader()
#34 /var/www/wiki/mediawiki/core/includes/cache/localisation/LocalisationCache.php(463): LocalisationCache->recache(string)
#35 /var/www/wiki/mediawiki/core/includes/cache/localisation/LocalisationCache.php(337): LocalisationCache->initLanguage(string)
#36 /var/www/wiki/mediawiki/core/includes/cache/localisation/LocalisationCache.php(278): LocalisationCache->loadItem(string, string)
#37 /var/www/wiki/mediawiki/core/languages/Language.php(4530): LocalisationCache->getItem(string, string)
#38 /var/www/wiki/mediawiki/core/languages/Language.php(265): Language::getFallbacksFor(string)
#39 /var/www/wiki/mediawiki/core/languages/Language.php(226): Language::newFromCode(string)
#40 /var/www/wiki/mediawiki/core/includes/ServiceWiring.php(160): Language::factory(string)
#41 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#42 /var/www/wiki/mediawiki/core/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#43 /var/www/wiki/mediawiki/core/includes/MediaWikiServices.php(537): Wikimedia\Services\ServiceContainer->getService(string)
#44 /var/www/wiki/mediawiki/core/includes/Setup.php(800): MediaWiki\MediaWikiServices->getContentLanguage()
#45 /var/www/wiki/mediawiki/core/includes/WebStart.php(81): require_once(string)
#46 /var/www/wiki/mediawiki/core/index.php(39): require(string)
#47 {main}

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Tagging MW-1.34-release as we should ideally get a fix (whether a new patch, or a revert) into 1.34

daniel added a subscriber: daniel.

Bumping back to inbox, since this got tagged for the 1.34 release, which makes it urgent.

This assumption is broken by the fact that sometimes Language.php calls into LocalisationCache before it has finished constructing the Language object:

Ideally, LocalizationCache would not depend on the Language class, so this call would be fine.

Frame 38 is where it gets the fallback list, which I suspect is a step we didn't think about.

Language, or, in the future, LanguageFactory, will need to be able to get the fallback list from LocalisationCache. Which is OK if LocalisationCache no longer depends on Language.

@Simetrical Does your work on the LanguageFactory service change/improve this?

Yes, https://gerrit.wikimedia.org/r/c/mediawiki/core/+/532680 pulls the static methods out of the Language class, allowing LocalizationCache to use them without any knowledge of Language and LanguageFactory.

I'm about to pick this up again, I could use some help with reviewing the chain of patches starting at https://gerrit.wikimedia.org/r/c/mediawiki/core/+/532679/12

Another surprising part is BlobStoreFactory requiring a ContentLanguage:

  • #20 MediaWikiGadgetsDefinitionRepo.php:139: Revision::newFromTitle(Title)
  • #19 Revision.php:138: Revision::getRevisionLookup()
  • #14 ServiceWiring.php:611: MediaWiki\MediaWikiServices->getRevisionStore()
  • #10 ServiceWiring.php:625: MediaWiki\MediaWikiServices->getRevisionStoreFactory()
  • #6 ServiceWiring.php:632: MediaWiki\MediaWikiServices->getBlobStoreFactory()
  • #2 ServiceWiring.php:10: MediaWiki\MediaWikiServices->getContentLanguage()

I wonder what that's all about.

It's about the legacy encoding - when applying it, we need to language for context. We can restructure the code so this call only happens on wikis where a legacy encoding is actually set, but that would not remove the fundamental problem.

Once we have a LanguageFactory service, we can introduce an EncodingConverter that has the encoding, the language code, and the LanguageFactory, and gets the Language object only when needed. Which will only be the case when loading very old revisions.

If it were possible to clear the MessageBlobStore without constructing a ResourceLoader, that would solve this circular dependency problem.

This is a nasty tangle. Constructing a Language causes LocalisationCache::recache() to be called. LocalisationCache::recache() needs to be able to call MessageBlobStore::clear(). The MessageBlobStore instance is owned by the ResourceLoader.

If we could manage MessageBlobStore as a standalone service and inject it into ResourceLoader, that would solve the problem. But MessageBlobStore needs to know the ResourceLoader as well, the two are paired together: MessageBlobStore calls ResourceLoader::getModulesByMessage().

The only way I see is to extract a ResourceModuleManager from ResourceLoader, so that ResourceLoader can depend on MessageBlobStore and ResourceModuleManager, and MessageBlobStore also depends on ResourceModuleManager.

And all this because we initialized the Language object from LocalisationCache, and that initialization may cause a recache(), which may purge cached messages elsewhere... The entire mechanism seems flawed.

Pinging @Nikerabbit for ideas.

To clarify, there is no recursion in the default construction of Language, LocalisationCache, and ResourceLoader.

The recursion happens because the Gadgets extension needs to fetch wiki page content (MediaWiki:Gadgets-definition) and BlobStore constructing the ContentLanguage. That is another place where a tie could perhaps be broken. The BlobStore needs two things: config key LegacyEncoding and to call the native php function iconv() which we proxy in Language::iconv (not sure why?), with this never overridden or otherwise language-specific.

Change 540163 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] [WIP] Storage: SqlBlobStore no longer needs Lanugage object

https://gerrit.wikimedia.org/r/540163

daniel added a comment.Oct 1 2019, 7:12 PM

Change 540163 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] [WIP] Storage: SqlBlobStore no longer needs Lanugage object

Much simpler solution than what I had in mind, thanks :)

Krinkle added a comment.EditedOct 1 2019, 7:14 PM

Cool. Note, I won't be able to work on this further, so you may want to take over amending it or re-upload. Sorry :/

Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.

Change 540163 merged by jenkins-bot:
[mediawiki/core@master] Storage: SqlBlobStore no longer needs Lanugage object

https://gerrit.wikimedia.org/r/540163

Pchelolo closed this task as Resolved.Oct 2 2019, 4:47 PM
Pchelolo claimed this task.
ArielGlenn added a subscriber: ArielGlenn.EditedOct 16 2019, 4:37 PM

I saw this today on a local installation just updated to 1.35.0-wmf.1 with the patch. There's a slightly different stack trace:

#0 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#1 /home/system/www/html/elwv/includes/MediaWikiServices.php(539): Wikimedia\Services\ServiceContainer->getService(string)
#2 /home/system/www/html/elwv/includes/ServiceWiring.php(133): MediaWiki\MediaWikiServices->getContentLanguage()
#3 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#4 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#5 /home/system/www/html/elwv/includes/MediaWikiServices.php(507): Wikimedia\Services\ServiceContainer->getService(string)
#6 /home/system/www/html/elwv/includes/ServiceWiring.php(679): MediaWiki\MediaWikiServices->getCommentStore()
#7 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#8 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#9 /home/system/www/html/elwv/includes/MediaWikiServices.php(932): Wikimedia\Services\ServiceContainer->getService(string)
#10 /home/system/www/html/elwv/includes/ServiceWiring.php(668): MediaWiki\MediaWikiServices->getRevisionStoreFactory()
#11 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#12 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#13 /home/system/www/html/elwv/includes/MediaWikiServices.php(924): Wikimedia\Services\ServiceContainer->getService(string)
#14 /home/system/www/html/elwv/includes/ServiceWiring.php(654): MediaWiki\MediaWikiServices->getRevisionStore()
#15 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#16 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#17 /home/system/www/html/elwv/includes/MediaWikiServices.php(908): Wikimedia\Services\ServiceContainer->getService(string)
#18 /home/system/www/html/elwv/includes/Revision.php(76): MediaWiki\MediaWikiServices->getRevisionLookup()
#19 /home/system/www/html/elwv/includes/Revision.php(139): Revision::getRevisionLookup()
#20 /home/system/www/html/elwv/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(139): Revision::newFromTitle(Title)
#21 /home/system/www/html/elwv/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(108): MediaWikiGadgetsDefinitionRepo->fetchStructuredList()
#22 /home/system/www/html/elwv/includes/libs/objectcache/wancache/WANObjectCache.php(1424): MediaWikiGadgetsDefinitionRepo->{closure}(boolean, integer, array, NULL)
#23 /home/system/www/html/elwv/includes/libs/objectcache/wancache/WANObjectCache.php(1278): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array)
#24 /home/system/www/html/elwv/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(115): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#25 /home/system/www/html/elwv/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(31): MediaWikiGadgetsDefinitionRepo->loadGadgets()
#26 /home/system/www/html/elwv/extensions/Gadgets/includes/GadgetHooks.php(134): MediaWikiGadgetsDefinitionRepo->getGadgetIds()
#27 /home/system/www/html/elwv/includes/Hooks.php(174): GadgetHooks::registerModules(ResourceLoader)
#28 /home/system/www/html/elwv/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#29 /home/system/www/html/elwv/includes/ServiceWiring.php(640): Hooks::run(string, array)
#30 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#31 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#32 /home/system/www/html/elwv/includes/MediaWikiServices.php(892): Wikimedia\Services\ServiceContainer->getService(string)
#33 /home/system/www/html/elwv/includes/ServiceWiring.php(314): MediaWiki\MediaWikiServices->getResourceLoader()
#34 /home/system/www/html/elwv/includes/cache/localisation/LocalisationCache.php(1044): Wikimedia\Services\ServiceContainer->{closure}()
#35 /home/system/www/html/elwv/includes/cache/localisation/LocalisationCache.php(484): LocalisationCache->recache(string)
#36 /home/system/www/html/elwv/includes/cache/localisation/LocalisationCache.php(358): LocalisationCache->initLanguage(string)
#37 /home/system/www/html/elwv/includes/cache/localisation/LocalisationCache.php(299): LocalisationCache->loadItem(string, string)
#38 /home/system/www/html/elwv/languages/Language.php(4522): LocalisationCache->getItem(string, string)
#39 /home/system/www/html/elwv/languages/Language.php(263): Language::getFallbacksFor(string)
#40 /home/system/www/html/elwv/languages/Language.php(224): Language::newFromCode(string)
#41 /home/system/www/html/elwv/includes/ServiceWiring.php(162): Language::factory(string)
#42 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#43 /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#44 /home/system/www/html/elwv/includes/MediaWikiServices.php(539): Wikimedia\Services\ServiceContainer->getService(string)
#45 /home/system/www/html/elwv/includes/Setup.php(777): MediaWiki\MediaWikiServices->getContentLanguage()
#46 /home/system/www/html/elwv/includes/WebStart.php(81): require_once(string)
#47 /home/system/www/html/elwv/index.php(41): require(string)
#48 {main}

Here's the loop:

RuntimeException from line 448 of /home/system/www/html/elwv/includes/libs/services/ServiceContainer.php: Circular dependency when creating service! ContentLanguage -> ResourceLoader -> RevisionLookup -> RevisionStore -> RevisionStoreFactory -> CommentStore -> ContentLanguage
Krinkle reopened this task as Open.EditedOct 16 2019, 4:49 PM

Different trace indeed. This one is not triggered by RevisionStore's own need for ContentLanguage (that one was solved earlier in this task).

Formatted top-down:

  1. Setup.php: MediaWikiServices->getContentLanguage
  2. Language::factory > Language::newFromCode > LocalisationCache->recache
  3. MediaWikiServices->getResourceLoader
  4. GadgetHooks::registerModules > MediaWikiServices->getRevisionLookup
  5. MediaWikiServices->getRevisionStoreFactory > MediaWikiServices->getCommentStore > MediaWikiServices->getContentLanguage
WDoranWMF removed Pchelolo as the assignee of this task.Oct 23 2019, 6:42 PM
WDoranWMF removed a project: Core Platform Team.
WDoranWMF added a subscriber: Pchelolo.
Kghbln added a comment.Nov 6 2019, 9:07 AM

The upgrade to the latest release candidate did not help the cause. After invoking the "Gadgets" extension I still get the same error.

Setup

  • MediaWiki 1.34.0-rc.1 (6170b0c) 22:40, 5 November 2019
  • PHP 7.2.24-0ubuntu0.18.04.1 (apache2handler)
  • MariaDB 10.1.41-MariaDB-0ubuntu0.18.04.1

Issue

[b3a3fda3e18e7eca55796a5b] /wiki/Sp%C3%A9cial:Version RuntimeException from line 448 of /../w/includes/libs/services/ServiceContainer.php: Circular dependency when creating service! ContentLanguage -> ResourceLoader -> RevisionLookup -> RevisionStore -> RevisionStoreFactory -> BlobStoreFactory -> ContentLanguage

Backtrace

#0 /../w/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#1 /../w/includes/MediaWikiServices.php(540): Wikimedia\Services\ServiceContainer->getService(string)
#2 /../w/includes/ServiceWiring.php(113): MediaWiki\MediaWikiServices->getContentLanguage()
#3 /../w/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#4 /../w/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#5 /../w/includes/MediaWikiServices.php(459): Wikimedia\Services\ServiceContainer->getService(string)
#6 /../w/includes/ServiceWiring.php(723): MediaWiki\MediaWikiServices->getBlobStoreFactory()
#7 /../w/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#8 /../w/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#9 /../w/includes/MediaWikiServices.php(941): Wikimedia\Services\ServiceContainer->getService(string)
#10 /../w/includes/ServiceWiring.php(716): MediaWiki\MediaWikiServices->getRevisionStoreFactory()
#11 /../w/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#12 /../w/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#13 /../w/includes/MediaWikiServices.php(933): Wikimedia\Services\ServiceContainer->getService(string)
#14 /../w/includes/ServiceWiring.php(702): MediaWiki\MediaWikiServices->getRevisionStore()
#15 /../w/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#16 /../w/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#17 /../w/includes/MediaWikiServices.php(917): Wikimedia\Services\ServiceContainer->getService(string)
#18 /../w/includes/Revision.php(76): MediaWiki\MediaWikiServices->getRevisionLookup()
#19 /../w/includes/Revision.php(139): Revision::getRevisionLookup()
#20 /../w/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(139): Revision::newFromTitle(Title)
#21 /../w/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(108): MediaWikiGadgetsDefinitionRepo->fetchStructuredList()
#22 /../w/includes/libs/objectcache/wancache/WANObjectCache.php(1424): MediaWikiGadgetsDefinitionRepo->{closure}(boolean, integer, array, NULL)
#23 /../w/includes/libs/objectcache/wancache/WANObjectCache.php(1278): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array)
#24 /../w/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(115): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#25 /../w/extensions/Gadgets/includes/MediaWikiGadgetsDefinitionRepo.php(31): MediaWikiGadgetsDefinitionRepo->loadGadgets()
#26 /../w/extensions/Gadgets/includes/GadgetHooks.php(134): MediaWikiGadgetsDefinitionRepo->getGadgetIds()
#27 /../w/includes/Hooks.php(174): GadgetHooks::registerModules(ResourceLoader)
#28 /../w/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#29 /../w/includes/ServiceWiring.php(688): Hooks::run(string, array)
#30 /../w/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#31 /../w/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#32 /../w/includes/MediaWikiServices.php(901): Wikimedia\Services\ServiceContainer->getService(string)
#33 /../w/includes/ServiceWiring.php(323): MediaWiki\MediaWikiServices->getResourceLoader()
#34 /../w/includes/cache/localisation/LocalisationCache.php(1049): Wikimedia\Services\ServiceContainer->{closure}()
#35 /../w/includes/cache/localisation/LocalisationCache.php(489): LocalisationCache->recache(string)
#36 /../w/includes/cache/localisation/LocalisationCache.php(363): LocalisationCache->initLanguage(string)
#37 /../w/includes/cache/localisation/LocalisationCache.php(304): LocalisationCache->loadItem(string, string)
#38 /../w/languages/Language.php(4413): LocalisationCache->getItem(string, string)
#39 /../w/languages/Language.php(265): Language::getFallbacksFor(string)
#40 /../w/languages/Language.php(225): Language::newFromCode(string)
#41 /../w/includes/ServiceWiring.php(164): Language::factory(string)
#42 /../w/includes/libs/services/ServiceContainer.php(458): Wikimedia\Services\ServiceContainer->{closure}(MediaWiki\MediaWikiServices)
#43 /../w/includes/libs/services/ServiceContainer.php(427): Wikimedia\Services\ServiceContainer->createService(string)
#44 /../w/includes/MediaWikiServices.php(540): Wikimedia\Services\ServiceContainer->getService(string)
#45 /../w/includes/Setup.php(801): MediaWiki\MediaWikiServices->getContentLanguage()
#46 /../w/includes/WebStart.php(81): require_once(string)
#47 /../w/index.php(41): require(string)
#48 {main}
Reedy added a comment.Nov 6 2019, 4:47 PM

The upgrade to the latest release candidate did not help the cause. After invoking the "Gadgets" extension I still get the same error.

Bleugh. It looks like https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/540163/ wasn't backported. Thanks for the headsup

Change 549114 had a related patch set uploaded (by Reedy; owner: Krinkle):
[mediawiki/core@REL1_34] Storage: SqlBlobStore no longer needs Language object

https://gerrit.wikimedia.org/r/549114

Change 549114 merged by jenkins-bot:
[mediawiki/core@REL1_34] Storage: SqlBlobStore no longer needs Language object

https://gerrit.wikimedia.org/r/549114

Reedy added a comment.Nov 6 2019, 6:11 PM

The upgrade to the latest release candidate did not help the cause. After invoking the "Gadgets" extension I still get the same error.

Bleugh. It looks like https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/540163/ wasn't backported. Thanks for the headsup

At least my text on the announcement was right (and we weren't claiming it was fixed!)

There is a known and intermittent issue that surfaces itself as "Circular dependency when creating service!", see T231866 [3] for more information. If you experience this issue, please let us know (on the ticket) to get an idea of how prevalent it is. It is intended to get this resolved for 1.34.0.

Kghbln added a comment.Nov 8 2019, 5:28 PM

Just updated to master of REL1_34 and invoked Gadets again. The first access to the wiki was still giving the reported error, not sure why, however every access to it afterwards was cool again. I will keep as is for now and report back in case of new worries. Anyways, thanks for the fix.

Reedy added a comment.Nov 8 2019, 6:00 PM

Just updated to master of REL1_34 and invoked Gadets again. The first access to the wiki was still giving the reported error, not sure why, however every access to it afterwards was cool again. I will keep as is for now and report back in case of new worries. Anyways, thanks for the fix.

There is another one intermittently about too

Kghbln added a comment.Nov 9 2019, 9:45 PM

There is another one intermittently about too

Ah, ok, so we still need the ultimate fix for these "circular dependency" issues. I will sit tight.

Reedy added a comment.Nov 9 2019, 9:46 PM

There is another one intermittently about too

Ah, ok, so we still need the ultimate fix for these "circular dependency" issues. I will sit tight.

Unfortunately :)

Different trace indeed. This one is not triggered by RevisionStore's own need for ContentLanguage (that one was solved earlier in this task).
Formatted top-down:

  1. Setup.php: MediaWikiServices->getContentLanguage
  2. Language::factory > Language::newFromCode > LocalisationCache->recache
  3. MediaWikiServices->getResourceLoader
  4. GadgetHooks::registerModules > MediaWikiServices->getRevisionLookup
  5. MediaWikiServices->getRevisionStoreFactory > MediaWikiServices->getCommentStore > MediaWikiServices->getContentLanguage
kchapman added a subscriber: kchapman.

@daniel would you be able to look at this at some point?

There is another one intermittently about too

Ah, ok, so we still need the ultimate fix for these "circular dependency" issues. I will sit tight.

The ultimate fix is to make sure that things don't depend on each other in circular ways... There is no easy solution for this, there will have to be a separate solution for every instance of such a circular dependency.

 Setup.php: MediaWikiServices->getContentLanguage
Language::factory > Language::newFromCode > LocalisationCache->recache
MediaWikiServices->getResourceLoader
GadgetHooks::registerModules > MediaWikiServices->getRevisionLookup
MediaWikiServices->getRevisionStoreFactory > MediaWikiServices->getCommentStore > MediaWikiServices->getContentLanguage

Seems like the easiest way to cut this loop is to avoid the instantiation of a ResourceLoader when LocalisationCache->recache() is called. We only need the ResourceLoader so we can get a MessageBlobStore, and we only need that MessageBlobStore to call the clear() method on it, which is implemented as:

	public function clear() {
		$cache = $this->wanCache;
		$cache->touchCheckKey( $cache->makeGlobalKey( __CLASS__ ), $cache::HOLDOFF_TTL_NONE );
	}

This could in theory be a static method which takes the cache object as a parameter. $this->wanCache is currently hard-coded in the constructor of MessageBlobStore to be the default WANObjectCache.

This would be an easy fix, but it feels a bit brittle. Are we sure that this will always work the same? What are the chances that the logic will change in a way that requires a specific instance of MessageBlobStore and ResourceLoader?

Note that the static version of MessageBlobStore::clear would be called by a callback created in ServiceWiring.php, where we have knowledge about all the service instances, and probably also about what cache object gets injected where. But it's not pretty...

Does this sound like a decent solution? Alternative ideas?

Change 553170 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] DNM: LocalisationCache: don't instantiate ResourceLoader

https://gerrit.wikimedia.org/r/553170

Zerbey added a subscriber: Zerbey.

Change 553170 merged by jenkins-bot:
[mediawiki/core@master] LocalisationCache: Don't instantiate ResourceLoader

https://gerrit.wikimedia.org/r/553170

Change 553868 had a related patch set uploaded (by Reedy; owner: Daniel Kinzler):
[mediawiki/core@REL1_34] LocalisationCache: Don't instantiate ResourceLoader

https://gerrit.wikimedia.org/r/553868

Change 553868 merged by jenkins-bot:
[mediawiki/core@REL1_34] LocalisationCache: Don't instantiate ResourceLoader

https://gerrit.wikimedia.org/r/553868

Reedy added a comment.EditedDec 3 2019, 2:30 AM

Do we know if anything else is outstanding here?

It seems to have fixed the errors I was getting on my dev wiki on a cold load.... @ArielGlenn does it seem fixed for you too? (though it's not in a wmf branch yet - not sure if it's worth backporting)

Do we know if anything else is outstanding here?
It seems to have fixed the errors I was getting on my dev wiki on a cold load.... @ArielGlenn does it seem fixed for you too? (though it's not in a wmf branch yet - not sure if it's worth backporting)

I cherry-picked https://gerrit.wikimedia.org/r/553170 against 1.35-wmf.1 and I now get a 500 error and no php stack trace to the log, just blank content. I should test this against wmf.8, probably not til the end of the week though.

Do we know if anything else is outstanding here?
It seems to have fixed the errors I was getting on my dev wiki on a cold load.... @ArielGlenn does it seem fixed for you too? (though it's not in a wmf branch yet - not sure if it's worth backporting)

I cherry-picked https://gerrit.wikimedia.org/r/553170 against 1.35-wmf.1 and I now get a 500 error and no php stack trace to the log, just blank content. I should test this against wmf.8, probably not til the end of the week though.

Did you have time to do this?

Sigh.. no. adding to my todo list.

I guess this is one of the very few bugs blocking the upcoming November release of 1.34.0

I guess this is one of the very few bugs blocking the upcoming November release of 1.34.0

Isn't this fixed in 1.34? The path was backported, see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/553868.

I guess this is one of the very few bugs blocking the upcoming November release of 1.34.0

Isn't this fixed in 1.34? The path was backported, see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/553868.

One report yes, one report no.

I guess this is one of the very few bugs blocking the upcoming November release of 1.34.0

Isn't this fixed in 1.34? The path was backported, see https://gerrit.wikimedia.org/r/c/mediawiki/core/+/553868.

One report yes, one report no.

Ariel's report of a 500 sounds like something went astray with the cherry pick. If it was that broken on master (or REL1_34), we'd definitely know about it

I certainly can't replicate it even though I was an original reporter..... Hopefully that's for the best ;)

Jdforrester-WMF closed this task as Resolved.Dec 19 2019, 4:54 PM

Provisionally declaring fixed, then.

Finally (!) got 1.35.0-wmf.11 set up locally and all is well. Thanks everyone!