Can reproduce on beta cluster as well: https://en.wikipedia.beta.wmflabs.org/wiki/Topic:Vp9cnfpdti9bnda2

Mentioned in SAL (#wikimedia-operations) [2020-06-30T14:43:45Z] <hashar> Train blocked on Flow being broken: T256761 # T254176

They seem tightly related and definitely cause StructuredDiscussions to explode. One stacktrace originates from ElectronPdfServiceHooks::onSidebarBeforeOutput

Both stacktraces refers to the building the sidebar code which is in Vector (legacy skin) , Electron might not be the root cause.

This has vibes of T255934: ElectronPDFs should move Printable version link to its own section

I think this may be T255934. I'd suggest reverting https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ElectronPdfService/+/607087/7/src/ElectronPdfServiceHooks.php#57

Change 608649 had a related patch set uploaded (by Aron Manning; owner: Aron Manning):
[mediawiki/extensions/ElectronPdfService@master] Hotfix: "Undefined index: print"

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ElectronPdfService/ /608649

>>> $t = Title::newFromText('Topic:Vp9cnfpdti9bnda2')
>>> $skin = \MediaWiki\MediaWikiServices::getInstance()->getSkinFactory()->makeSkin('vector')
>>> $skin->getContext()->setTitle($t)
>>> $skin->buildSidebar()

=> [
     "navigation" => [
       [
         "text" => "Main page",
         "href" => "/wiki/Main_Page",
         "id" => "n-mainpage-description",
         "active" => false,
       ],
       [
         "text" => "Contents",
         "href" => "/wiki/Wikipedia:Contents",
         "id" => "n-contents",
         "active" => false,
       ],
       [
         "text" => "Current events",
         "href" => "/wiki/Portal:Current_events",
         "id" => "n-currentevents",
         "active" => false,
       ],
       [
         "text" => "Random article",
         "href" => "/wiki/Special:Random",
         "id" => "n-randompage",
         "active" => false,
       ],
       [
         "text" => "About Wikipedia",
         "href" => "/wiki/Wikipedia:About",
         "id" => "n-aboutsite",
         "active" => false,
       ],
       [
         "text" => "Contact page",
         "href" => "/wiki//wiki/Wikipedia:Contact_us",
         "id" => "n-contactpage",
         "active" => false,
       ],
       [
         "text" => "Donate to Wikipedia",
         "href" => "//wikimediafoundation.org/wiki/Special:Landingcheck?landing_page=WMFJA085&language=en&utm_source=donate&utm_medium=sidebar&utm_campaign=20101204SB002",
         "id" => "n-sitesupport",
         "active" => false,
         "rel" => "nofollow",
       ],
       [
         "text" => "Wikipedia store",
         "href" => "//shop.wikimedia.org",
         "title" => "Visit the Wikipedia store",
         "id" => "n-shoplink",
       ],
     ],
     "SEARCH" => [],
     "interaction" => [
       [
         "text" => "Help",
         "href" => "https://www.mediawiki.org/wiki/Special:MyLanguage/Help:Contents",
         "id" => "n-help",
         "active" => false,
       ],
       [
         "text" => "Community portal",
         "href" => "/wiki/Wikipedia:Community_portal",
         "id" => "n-portal",
         "active" => false,
       ],
       [
         "text" => "Recent changes",
         "href" => "/wiki/Special:RecentChanges",
         "id" => "n-recentchanges",
         "active" => false,
       ],
       [
         "text" => "Upload file",
         "href" => "/wiki/Wikipedia:File_Upload_Wizard",
         "id" => "n-upload",
         "active" => false,
       ],
     ],
     "TOOLBOX" => [
       "whatlinkshere" => [
         "href" => "/wiki/Special:WhatLinksHere/Topic:Vp9cnfpdti9bnda2",
         "id" => "t-whatlinkshere",
       ],
       "recentchangeslinked" => [
         "href" => "/wiki/Special:RecentChangesLinked/Topic:Vp9cnfpdti9bnda2",
         "msg" => "recentchangeslinked-toolbox",
         "id" => "t-recentchangeslinked",
         "rel" => "nofollow",
       ],
       "upload" => [
         "href" => "/wiki/Wikipedia:File_Upload_Wizard",
         "id" => "t-upload",
       ],
       "specialpages" => [
         "href" => "/wiki/Special:SpecialPages",
         "id" => "t-specialpages",
       ],
       "info" => [
         "text" => "Page information",
         "href" => "/w/index.php?title=Topic:Vp9cnfpdti9bnda2&action=info",
         "id" => "t-info",
       ],
     ],
     "LANGUAGES" => [],
     "electronpdfservice-sidebar-portlet-heading" => [
       [
         "text" => "Download as PDF",
         "id" => "electron-print_pdf",
         "href" => "/w/index.php?title=Special:ElectronPdf&page=Topic%3AVp9cnfpdti9bnda2&action=show-download-screen",
       ],
       null,
     ],
   ]

This might just be a debugging artifact, as I'm probably skipping some skin initialization steps, but note the extra null in the end. That's consistent with the first error.

In T256761#6268047, @Ammarpad wrote:

I think we don't need to revert that. I suspect the print key is not present in flow page for some reasons. Need to check properly though.

No need. Above patch will solve it at least in the short term.

...that null is caused by the print field not existing in ElectronPdfServiceHooks::onSidebarBeforeOutput(). So the first error is just a side-effect of the second one.

master and wmf/1.35.0-wmf.39 branches are at the same commit. How do you do backporting in this case? Just fast-forward wmf.39?

In T256761#6268123, @Demian wrote:

master and wmf/1.35.0-wmf.39 branches are at the same commit. How do you do backporting in this case? Just fast-forward wmf.39?

They won't be at the same commit once the fix gets merged.

Do I submit a separate patch for that branch, or it can be fast-forwarded (rebased) to the same commit?

In T256761#6268135, @Demian wrote:

Do I submit a separate patch for that branch, or it can be fast-forwarded (rebased) to the same commit?

Submit your fix to master and then it can be backported to wmf.39

Change 608485 had a related patch set uploaded (by Aron Manning; owner: Aron Manning):
[mediawiki/extensions/ElectronPdfService@wmf/1.35.0-wmf.39] Hotfix: "Undefined index: print"

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ElectronPdfService/ /608485

In T256761#6268135, @Demian wrote:

Do I submit a separate patch for that branch, or it can be fast-forwarded (rebased) to the same commit?

You can cherry-pick it to another branch via the Gerrit UI. I don't think there's a similarly easy way for fast-forwarding (you could redefine the branch tip but that requires advanced privileges).

Thanks! Done.
Thanks @RhinosF1 for the quick notice.

Change 608649 merged by jenkins-bot:
[mediawiki/extensions/ElectronPdfService@master] Hotfix: "Undefined index: print"

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ElectronPdfService/ /608649

In T256761#6267868, @Majavah wrote:

Can reproduce on beta cluster as well: https://en.wikipedia.beta.wmflabs.org/wiki/Topic:Vp9cnfpdti9bnda2

Still broken, but maybe that's just due to the sidebar cache?

In T256761#6268300, @Tgr wrote:

In T256761#6267868, @Majavah wrote:

Can reproduce on beta cluster as well: https://en.wikipedia.beta.wmflabs.org/wiki/Topic:Vp9cnfpdti9bnda2

Still broken, but maybe that's just due to the sidebar cache?

Another thread is working (https://en.wikipedia.beta.wmflabs.org/wiki/Topic:Vp9iok8ssazd6a62). I'd say it is due to caching.

@Demian do you want to deploy the backport? The next window is in an hour. See https://wikitech.wikimedia.org/wiki/Backport_windows for docs.
(I can do it if aren't interested.)

Sure, thanks for the link

I'm not on IRC, is that a hard requirement in this case? If so, I'd like to ask you.

Yeah, deployments are coordinated on IRC. Will do then. Thanks for writing the patch!

Thank you!

Thank you for all the investigation! I was in meeting and I am heading to dinner.

I will be back at 19:00 UTC which is the slot "Mediawiki train - European+American Version (secondary timeslot)"

Change 608485 merged by jenkins-bot:
[mediawiki/extensions/ElectronPdfService@wmf/1.35.0-wmf.39] Hotfix: "Undefined index: print"

https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ElectronPdfService/ /608485

Mentioned in SAL (#wikimedia-operations) [2020-06-30T18:23:17Z] <tgr@deploy1001> Synchronized php-1.35.0-wmf.39/extensions/ElectronPdfService/src/ElectronPdfServiceHooks.php: Backport: [[gerrit:608485|Hotfix: "Undefined index: print" (T256761)]] (duration: 01m 05s)

Working on testwiki: https://test.wikipedia.org/wiki/Topic:Vp9iyl4amw7ofe4n

Thanks everyone who worked on solving this!

I am impressed :] I will promote the wikis again in a minute. Thank you very much.