Page MenuHomePhabricator
Create Task
Maniphest T175080

Flow fails to load content when running CirrusSearchLinksUpdate jobs
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Spotted in production:

[09a78b5c2ff178424bb3103e] /rpc/RunJobs.php?wiki=testwiki&type=cirrusSearchLinksUpdate&maxtime=30&maxmem=300M   Flow\Exception\InvalidDataException from line 366 of /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Model/AbstractRevision.php: Failed to load the content

With the following stacktrace:

#0 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Templating.php(154): Flow\Model\AbstractRevision->getContent(string)
#1 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Formatter/RevisionFormatter.php(1013): Flow\Templating->getContent(Flow\Model\PostRevision, string)
#2 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Formatter/RevisionFormatter.php(334): Flow\Formatter\RevisionFormatter->processParam(string, Flow\Model\PostRevision, Flow\Model\UUID, Flow\View, Flow\Formatter\TopicRow)
#3 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Formatter/TopicFormatter.php(45): Flow\Formatter\RevisionFormatter->formatApi(Flow\Formatter\TopicRow, Flow\View)
#4 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Block/Topic.php(642): Flow\Formatter\TopicFormatter->formatApi(Flow\Model\Workflow, array, Flow\View)
#5 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Block/Topic.php(549): Flow\Block\TopicBlock->renderTopicApi(array)
#6 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/View.php(189): Flow\Block\TopicBlock->renderApi(array)
#7 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/View.php(68): Flow\View->buildApiResponse(Flow\WorkflowLoader, array, string, array)
#8 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Content/BoardContent.php(220): Flow\View->show(Flow\WorkflowLoader, string)
#9 /srv/mediawiki/php-1.30.0-wmf.17/extensions/Flow/includes/Content/BoardContent.php(171): Flow\Content\BoardContent->generateHtml(Title, User)
#10 /srv/mediawiki/php-1.30.0-wmf.17/includes/content/ContentHandler.php(1205): Flow\Content\BoardContent->getParserOutput(Title, integer, ParserOptions)
#11 /srv/mediawiki/php-1.30.0-wmf.17/extensions/CirrusSearch/includes/Updater.php(368): ContentHandler->getParserOutputForIndexing(WikiPage, ParserCache)
#12 /srv/mediawiki/php-1.30.0-wmf.17/extensions/CirrusSearch/includes/Updater.php(199): CirrusSearch\Updater->buildDocumentsForPages(array, integer)
#13 /srv/mediawiki/php-1.30.0-wmf.17/extensions/CirrusSearch/includes/Updater.php(82): CirrusSearch\Updater->updatePages(array, integer)
#14 /srv/mediawiki/php-1.30.0-wmf.17/extensions/CirrusSearch/includes/Job/LinksUpdate.php(52): CirrusSearch\Updater->updateFromTitle(Title)
#15 /srv/mediawiki/php-1.30.0-wmf.17/extensions/CirrusSearch/includes/Job/Job.php(98): CirrusSearch\Job\LinksUpdate->doJob()
#16 /srv/mediawiki/php-1.30.0-wmf.17/includes/jobqueue/JobRunner.php(295): CirrusSearch\Job\Job->run()
#17 /srv/mediawiki/php-1.30.0-wmf.17/includes/jobqueue/JobRunner.php(193): JobRunner->executeJob(CirrusSearch\Job\LinksUpdate, Wikimedia\Rdbms\LBFactoryMulti, BufferingStatsdDataFactory, integer)
#18 /srv/mediawiki/rpc/RunJobs.php(47): JobRunner->run(array)
#19 {main}

It's impossible to tell from the trace (which I'll also remedy), but we think this is T95580: Flow data missing on Wikimedia production wikis. They're testwiki posts, but we need to make sure it doesn't break here.

Related Objects
Search...

Event Timeline

demon created this task.Sep 5 2017, 9:36 PM
Restricted Application added projects: Collaboration-Team-Triage, Discovery-ARCHIVED, Discovery-Search. · View Herald TranscriptSep 5 2017, 9:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
demon added a parent task: T170635: 1.30.0-wmf.17 deployment blockers.Sep 5 2017, 10:08 PM
Jdforrester-WMF added a subscriber: Jdforrester-WMF.Sep 6 2017, 7:45 PM
Jdforrester-WMF added a comment.Sep 6 2017, 7:48 PM

On hearing of this I initially presumed that T172025: Flow isAllowed gets actual revision text before it is needed might be related, but it wasn't merged. https://gerrit.wikimedia.org/r/#/q/project:mediawiki/extensions/Flow+is:merged doesn't show anything suggestive, and neither does https://gerrit.wikimedia.org/r/#/q/is:merged+project:mediawiki/extensions/CirrusSearch . Ideas?

Catrope added a subscriber: Catrope.Sep 6 2017, 8:01 PM

Does anyone know which wikis this error happens on? For example, does it only happen on testwiki and test2wiki?

EBernhardson added a subscriber: EBernhardson.Sep 6 2017, 8:03 PM

It was only deployed to group0, looks like mostly testwiki: https://logstash.wikimedia.org/goto/39be514a7e1e4ac2ef026fb846efaf83

Catrope added a comment.Sep 6 2017, 8:04 PM

Aha. But there are some fawiki occurrences too, although they look different.

• Mattflaschen-WMF claimed this task.Sep 6 2017, 8:55 PM
• Mattflaschen-WMF updated the task description. (Show Details)Sep 7 2017, 3:23 AM
• Mattflaschen-WMF closed this task as Invalid.EditedSep 7 2017, 3:54 AM

I still think the cause is as stated, but this is being caught, not impacting anything, and I don't think it's a release blocker.

An example is https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2017.09.05/mediawiki?id=AV5T7G-aTqnoMLVnze58&_g=h@44136fa .

We are catching it then logging it, which is tracked via caught_by.

I am still changing how this works for the benefit of the dumps (T139791). That code path does not go throw Templating, so it's actually not caught.

However, I don't think this is a bug.

And I believe the checks for release-blockers should exclude caught_by: other

• Mattflaschen-WMF updated the task description. (Show Details)Sep 7 2017, 5:29 AM
• Mattflaschen-WMF added a comment.Sep 7 2017, 6:50 AM

See https://gerrit.wikimedia.org/r/376478

demon reopened this task as Open.Sep 7 2017, 4:15 PM

It is a release blocker because of the sheer volume it's logged. It would drown out everything if I move forward.

debt removed a project: Discovery-Search.Sep 7 2017, 5:09 PM
Ladsgroup added a subscriber: Ladsgroup.Sep 7 2017, 6:35 PM
demon added a comment.Sep 7 2017, 7:10 PM

Er, I got this confused. Yes we still need to get the fix merged but it wasn't as loud as I thought.

• jmatazzoni edited projects, added Collaboration-Team-Triage (Collab-Team-This-Quarter); removed Collaboration-Team-Triage.Oct 9 2017, 8:03 PM
• jmatazzoni moved this task from Untriaged to QA Review on the Collaboration-Team-Triage (Collab-Team-This-Quarter) board.
Etonkovidova added a subscriber: Etonkovidova.Oct 13 2017, 8:10 PM

Checked the logstash for this type of erros - as far as I could see, there are no more such errors present.

Etonkovidova closed this task as Resolved.Oct 13 2017, 8:10 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL