Thanks for reporting this.

Correct pages like https://he.wikipedia.org/w/index.php?title=מיוחד:גרסה&uselang=en&debug=true include this CSS (which is missing on the broken pages):

#pt-notifications-notice .mw-echo-notifications-badge:before {
  background-image: url(data:image/svg+xml,%3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%0A%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20width%3D%2224%22%20height%3D%2224%22%20viewBox%3D%220%200%2024%2024%22%3E%0A%20%20%20%20%3Cpath%20d%3D%22M3%2013.35l1.8-7.2c.2-.996.81-1.8%201.8-1.8h10.8c.99%200%201.6.867%201.8%201.8l1.8%207.2v4.5c0%20.99-.81%201.8-1.8%201.8H4.8c-.99%200-1.8-.81-1.8-1.8v-4.5zm6.96%201.8h4.08c-.49.557-1.212.9-2.04.9a2.68%202.68%200%200%201-2.04-.9h4.08c.414-.472.66-1.098.66-1.8h4.14l-1.44-7.2H6.6l-1.44%207.2H9.3c0%20.702.246%201.328.66%201.8z%22%2F%3E%0A%3C%2Fsvg%3E%0A);
}

which is included from the URL https://he.wikipedia.org/w/load.php?debug=true&lang=en&modules=ext.echo.styles.badge&only=styles&skin=vector

Going to the broken page https://he.wikipedia.org/wiki/Special:GadgetUsage?uselang=en&debug=true and looking at the "Network" tab in your browser's developer tools, the URL https://he.wikipedia.org/w/load.php?debug=true&lang=en&modules=ext.echo.styles.badge&only=styles&skin=vector is not listed (=loaded).

For some reason a LOT of modules are missing on Special:GadgetUsage. I think this might be a bug in the special page, not in Echo. Investigating.

About regression: I'm 95% sure it was OK 19 days ago on this page.

In T173411#3529276, @IKhitron wrote:

About regression: I'm 95% sure it was OK 19 days ago on this page.

Thanks for that data point. Nothing notable has changed in the Gadgets extension in the last few weeks though. I looked at SpecialGadgetUsage.php and didn't see anything suspicious, so maybe this is an MW core issue. I can confirm that Special:GadgetUsage has the same bug on enwiki, but I haven't found any other special pages that are broken; I tested some other special pages that are also QueryPage subclasses and they worked, and Special:Gadgets (the other special page in the same extension) works too. @IKhitron do you by any chance know of any other special pages with this bug, or is it only Special:GadgetUsage?

I didn't see other pages kinds. And it's not only en, also fr and ru, and looks like all.

Btw, maybe it will help you. Our wiki has a third icon, a star, for our purposes, that copies at the runtime the code and design from the regular two and only changes the icon file address. And it works fine on this page.

Haven't been able to reproduce locally.

On what wiki, @kaldari?

A similar issue is reproducible on Special:Upload on test2.wp (https://test2.wikipedia.org/wiki/Special:Upload?safemode=1), but not on any others pages I tested.

We originally noticed this when testing Timeless after the deployment to test wikis (T154371), since it looks a lot more broken in that skin (T173525):

@matmarex, did you see some more names there? I checked hem on our wiki, and they are OK. It's something non deterministic?

Special:Upload is broken in this way on https://test.wikipedia.org/, https://test2.wikipedia.org/ and https://www.mediawiki.org/, but apparently not on he.wp (or other production wikis I checked). Perhaps this depends on some configuration too.

On what wiki, @kaldari?

My local wiki.

In T173411#3534087, @kaldari wrote:

Also reported for Special:Notifications.

That only seems to be true for some wikis and not others. See https://es.wikipedia.org/wiki/Especial:Notificaciones, https://fr.wikipedia.org/wiki/Spécial:Notifications, https://he.wikipedia.org/wiki/מיוחד:הודעות.

Per T173392 (dupe of this)

@Krenair wrote:

Timo looked into this with me during the hackathon, it seems BeforePageDisplay isn't getting run (?)

During the hackathon we investigated a bit and found the following:

• The module is not in the module queue when looking at the HTML source (e.g. the load.php request url in <link rel=stylesheet> is missing ext.echo.styles.badge).
• Browsing through MediaWiki source code, led me to believe that either it gets lost at run-time, or is never queued in the first place.
• Working backwards from where the stylesheet url is produced, the following (found by var_dump-ing on a debug server) are all missing the module at the time they are called:
  • ResourceLoaderClientHtml, OutputPage::getRlClient, OutputPage::headElement
• OutputPage::headElement is called as part of SkinTemplate::outputPage, which is called by OutputPage::output()
• OutputPage::output() runs hook BeforePageDisplay before calling $sk->outputPage.
• The module gets queued from Echo::beforePageDisplay (Echo:/Hooks.php#L847-L856)

We found that on wiki pages, it works fine. And on special pages it did not. However, then we found that it did work on a small subset of special pages.

• Wiki pages: OK
• Special:Blankpage: OK
• Special:SpecialPages: OK
• Special:Wantedfiles: OK
• Special:Notifications: Missing
• Special:Upload: Missing
• Special:Book: Missing

• OutputPage::output() is called from MediaWiki.php#main(). I found that on the pages where the module is missing, the line after $this->performRequest(); is reached, but the line before $output->output( is not.

At this point, I wasn't sure what further to debug aside from following down the whole inner stack of unrelated calls between performRequest and output (which I didn't feel like doing at the time).

Looking back now, I think that maybe I was wrong about it not calling output(). A more likely scenario is, in my opinion, that another extension's callback for BeforePageDisplay is returning false on some pages, causing Echo's handler not to be reached.

In T173411#3534174, @Niharika wrote:

That only seems to be true for some wikis and not others. See https://es.wikipedia.org/wiki/Especial:Notificaciones

\Wikibase\ClientHooks::onBeforePageDisplay
\Wikibase\ClientHooks::onBeforePageDisplayAddJsConfig
CharInsert::onBeforePageDisplay
GadgetHooks::beforePageDisplay
UrlShortenerHooks::onBeforePageDisplay
CentralAuthHooks::onBeforePageDisplay
GlobalCssJsHooks::onBeforePageDisplay
WikimediaMessagesHooks::onBeforePageDisplay
ElectronPdfServiceHooks::onBeforePageDisplay
MassMessageHooks::onBeforePageDisplay
MultimediaViewerHooks::getModulesForArticle
Popups\PopupsHooks::onBeforePageDisplay
MediaWiki\Linter\Hooks::onBeforePageDisplay
VisualEditorHooks::onBeforePageDisplay
WikiLoveHooks::onBeforePageDisplay
GuidedTourHooks::onBeforePageDisplay
MobileFrontendHooks::onBeforePageDisplay
FeaturedFeeds::beforePageDisplay
EchoHooks::beforePageDisplay
ThanksHooks::onBeforePageDisplay
GettingStarted\Hooks::onBeforePageDisplay
RelatedArticles\Hooks::onBeforePageDisplay
EventLoggingHooks::onBeforePageDisplay
JsonSchemaHooks::onBeforePageDisplay
WikimediaEventsHooks::onBeforePageDisplay
NavigationTimingHooks::onBeforePageDisplay
XAnalytics::onBeforePageDisplay
UniversalLanguageSelectorHooks::addModules
ContentTranslationHooks::addModules
WikimediaBadges\BeforePageDisplayHookHandler::onBeforePageDisplay
PageImages::onBeforePageDisplay
JsonConfig\JCSingleton::onBeforePageDisplay
ORES\Hooks::onBeforePageDisplay
CentralNoticeHooks::onBeforePageDisplay
FlaggedRevsUIHooks::onBeforePageDisplay
TimedMediaHandlerHooks::pageOutputHook

\Wikibase\ClientHooks::onBeforePageDisplay
\Wikibase\ClientHooks::onBeforePageDisplayAddJsConfig
CharInsert::onBeforePageDisplay
GadgetHooks::beforePageDisplay
UrlShortenerHooks::onBeforePageDisplay
CentralAuthHooks::onBeforePageDisplay
GlobalCssJsHooks::onBeforePageDisplay
WikimediaMessagesHooks::onBeforePageDisplay
ElectronPdfServiceHooks::onBeforePageDisplay
MassMessageHooks::onBeforePageDisplay
MultimediaViewerHooks::getModulesForArticle
Popups\PopupsHooks::onBeforePageDisplay
MediaWiki\Linter\Hooks::onBeforePageDisplay
VisualEditorHooks::onBeforePageDisplay
WikiLoveHooks::onBeforePageDisplay
GuidedTourHooks::onBeforePageDisplay
MobileFrontendHooks::onBeforePageDisplay
FeaturedFeeds::beforePageDisplay
EchoHooks::beforePageDisplay
ThanksHooks::onBeforePageDisplay
FlowHooks::onBeforePageDisplay
GettingStarted\Hooks::onBeforePageDisplay
RelatedArticles\Hooks::onBeforePageDisplay
EventLoggingHooks::onBeforePageDisplay
JsonSchemaHooks::onBeforePageDisplay
WikimediaEventsHooks::onBeforePageDisplay
NavigationTimingHooks::onBeforePageDisplay
XAnalytics::onBeforePageDisplay
UniversalLanguageSelectorHooks::addModules
ContentTranslationHooks::addModules
WikimediaBadges\BeforePageDisplayHookHandler::onBeforePageDisplay
PageImages::onBeforePageDisplay
JsonConfig\JCSingleton::onBeforePageDisplay
CentralNoticeHooks::onBeforePageDisplay
TimedMediaHandlerHooks::pageOutputHook

-enwiki
+eswiki
 \Wikibase\ClientHooks::onBeforePageDisplay
 \Wikibase\ClientHooks::onBeforePageDisplayAddJsConfig
 CentralAuthHooks::onBeforePageDisplay
@@ -9,7 +9,7 @@
 ElectronPdfServiceHooks::onBeforePageDisplay
 EventLoggingHooks::onBeforePageDisplay
 FeaturedFeeds::beforePageDisplay
-FlaggedRevsUIHooks::onBeforePageDisplay
+FlowHooks::onBeforePageDisplay
 GadgetHooks::beforePageDisplay
 GettingStarted\Hooks::onBeforePageDisplay
 GlobalCssJsHooks::onBeforePageDisplay
@@ -21,7 +21,6 @@
 MobileFrontendHooks::onBeforePageDisplay
 MultimediaViewerHooks::getModulesForArticle
 NavigationTimingHooks::onBeforePageDisplay
-ORES\Hooks::onBeforePageDisplay
 PageImages::onBeforePageDisplay
 Popups\PopupsHooks::onBeforePageDisplay
 RelatedArticles\Hooks::onBeforePageDisplay

	public static function onBeforePageDisplay( OutputPage &$out, Skin &$skin ) {
		if ( !self::oresUiEnabled( $out->getUser() ) ) {
			return;
		}

This might be our culprit. Although I was expecting to see a return false; here, given that, if I recall correctly, we changed it years ago that implicit return or null return defaults to true, not casted to false. And indeed, looking at Hooks::run() we only ever abort if either a string is returned, or explicit false (strict equality).

In T173411#3534560, @Krinkle wrote:

ORES:/Hooks.php

	public static function onBeforePageDisplay( OutputPage &$out, Skin &$skin ) {
		if ( !self::oresUiEnabled( $out->getUser() ) ) {
			return;
		}

This might be our culprit. Although I was expecting to see a return false; here, given that, if I recall correctly, we changed it years ago that implicit return or null return defaults to true, not casted to false. And indeed, looking at Hooks::run() we only ever abort if either a string is returned, or explicit false (strict equality).

I think that might be a red herring, because oresUiEnabled() returns true on all wikis for all users, due to the way our config is set up:

public static function oresUiEnabled( User $user ) {
        global $wgOresExtensionStatus, $wgOresUiEnabled;

        // Is the UI enabled or not?  If not, we've been deployed in
        // infrastructure-only mode, so hide all the UI elements.
        if ( !$wgOresUiEnabled ) {
                return false;
        }

        // enabled by default for everybody
        if ( $wgOresExtensionStatus === 'on' ) {
                return true;
        }

        // [...]
}

catrope@terbium:~$ mwscript eval.php hewiki
> var_dump($wgOresUiEnabled);
bool(true)

> var_dump($wgOresExtensionStatus);
string(2) "on"

(both of these are the default values set in extension.json, we don't override them in wmf-config anymore)

Also, why did you diff en against es? They have the same behavior on Special:Notifications (both work) for me, while fr is broken.

I can reproduce it from the command-line.

It's missing on Special:Upload with enwiki, but working with eswiki. And on Special:Blankpage, it's working with both.

$context = RequestContext::newExtraneousContext( SpecialPage::getTitleFor('Upload') )
$context->setUser( User::newSystemUser('MediaWiki default') );
$out = $context->getOutput(); $sk = $context->getSkin(); Hooks::run( 'BeforePageDisplay', [ &$out, &$sk ] );
return $out->getModuleStyles();
array(4) {
  string(21) "ext.charinsert.styles"
  string(28) "ext.gadget.charinsert-styles"
  string(27) "ext.globalCssJs.user.styles"
  string(27) "ext.globalCssJs.site.styles"
  # Echo is missing

> return $out->getModuleStyles();
array(10) {
  string(21) "ext.charinsert.styles"
  string(26) "ext.gadget.imagenesinfobox"
  string(27) "ext.globalCssJs.user.styles"
  string(27) "ext.globalCssJs.site.styles"
  string(46) "ext.visualEditor.desktopArticleTarget.noscript"
  string(21) "ext.echo.styles.badge" # OK
  string(19) "ext.echo.badgeicons"
  string(21) "ext.uls.interlanguage"
  string(19) "ext.wikimediaBadges"
  string(24) "ext.tmh.thumbnail.styles"

$context = RequestContext::newExtraneousContext( SpecialPage::getTitleFor('BlankPage') );
$context->setUser( User::newSystemUser('MediaWiki default') );
$out = $context->getOutput(); $sk = $context->getSkin(); Hooks::run( 'BeforePageDisplay', [ &$out, &$sk ] );
return $out->getModuleStyles();
array() {
"ext.gadget .." "ext.globalCssJs .." "ext.visualEditor .."
"ext.uls.interlanguage" "ext.wikimediaBadges"
"ext.echo.styles.badge" # OK
"ext.echo.badgeicons"

return $out->getModuleStyles();
array() {
"ext.gadget .." "ext.globalCssJs .." "ext.visualEditor .."
"ext.uls.interlanguage" "ext.wikimediaBadges"
"ext.echo.styles.badge" # OK
"ext.echo.badgeicons"

Given it's not based on which extension is installed (because the two hooks that differed in the registry, FlaggedRevs and ORES, were not the cause), @Catrope and I went through the list manually to find the handler that aborts the hook.

TimedMediaHandlerHooks::pageOutputHook
FlaggedRevsUIHooks::onBeforePageDisplay
CentralNoticeHooks::onBeforePageDisplay
ORES\Hooks::onBeforePageDisplay
JsonConfig\JCSingleton::onBeforePageDisplay
PageImages::onBeforePageDisplay
WikimediaBadges\BeforePageDisplayHookHandler::onBeforePageDisplay
ContentTranslationHooks::addModules
UniversalLanguageSelectorHooks::addModules
XAnalytics::onBeforePageDisplay
NavigationTimingHooks::onBeforePageDisplay
WikimediaEventsHooks::onBeforePageDisplay
JsonSchemaHooks::onBeforePageDisplay
EventLoggingHooks::onBeforePageDisplay
RelatedArticles\Hooks::onBeforePageDisplay
GettingStarted\Hooks::onBeforePageDisplay
ThanksHooks::onBeforePageDisplay
EchoHooks::beforePageDisplay
FeaturedFeeds::beforePageDisplay
MobileFrontendHooks::onBeforePageDisplay
GuidedTourHooks::onBeforePageDisplay
WikiLoveHooks::onBeforePageDisplay
VisualEditorHooks::onBeforePageDisplay
MediaWiki\Linter\Hooks::onBeforePageDisplay
Popups\PopupsHooks::onBeforePageDisplay
MultimediaViewerHooks::getModulesForArticle
MassMessageHooks::onBeforePageDisplay
ElectronPdfServiceHooks::onBeforePageDisplay
WikimediaMessagesHooks::onBeforePageDisplay
GlobalCssJsHooks::onBeforePageDisplay
CentralAuthHooks::onBeforePageDisplay
UrlShortenerHooks::onBeforePageDisplay
GadgetHooks::beforePageDisplay
CharInsert::onBeforePageDisplay
\Wikibase\ClientHooks::onBeforePageDisplayAddJsConfig
\Wikibase\ClientHooks::onBeforePageDisplay

At Popups\PopupsHooks::onBeforePageDisplay I found the following code, introduced by rEPOP29770a3ff7a7: Disable Previews on blacklisted pages.

	public static function onBeforePageDisplay( OutputPage &$out, Skin &$skin ) {
		$context = MediaWikiServices::getInstance()->getService( 'Popups.Context' );
		if ( $context->isTitleBlacklisted( $out->getTitle() ) ) {
			return false;
		}

Essentially, since this commit, on any page that is on the local wiki's Popups title blacklist, about half the extensions have their BeforePageDisplay hook disabled. The following were affected on enwiki: ContentTranslation, UniversalLanguageSelector, NavigationTiming, WikimediaEvents, EventLogging, and Echo. These likely were (partially) broken on the pages were Popups is disabled.

Change 372591 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Popups@master] Remove aborting of BeforePageDisplay hook

https://gerrit.wikimedia.org/r/372591

Change 372591 merged by jenkins-bot:
[mediawiki/extensions/Popups@master] Remove aborting of BeforePageDisplay hook

https://gerrit.wikimedia.org/r/372591

Change 372593 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/Popups@wmf/1.30.0-wmf.14] Remove aborting of BeforePageDisplay hook

https://gerrit.wikimedia.org/r/372593

Change 372593 merged by jenkins-bot:
[mediawiki/extensions/Popups@wmf/1.30.0-wmf.14] Remove aborting of BeforePageDisplay hook

https://gerrit.wikimedia.org/r/372593

Mentioned in SAL (#wikimedia-operations) [2017-08-22T23:10:32Z] <thcipriani@tin> Synchronized php-1.30.0-wmf.14/extensions/Popups/includes/PopupsHooks.php: SWAT: [[gerrit:372593|Remove aborting of BeforePageDisplay hook]] T173411 (duration: 00m 49s)

A sincere thank you to all involved in fixing this!

Cool, thanks.