NewPagesFeed not working when sorted oldest first
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Force_Radical
	Feb 21 2019, 4:31 AM

Description

NewPagesFeed does not display any articles when sorted Oldest first.

Steps to reproduce

Navigate to Special:NewPagesFeed on en.wikipedia.org
Tap on the "Oldest" radio button near the top right

Expected behavior : The NewPagesFeed displays articles oldest first
Current behavior : The articles in the feed disappear and nothing occurs

Notes

Console displays the following new error :

Uncaught ReferenceError: author_byline_html is not defined
    at child.eval (eval at _.template (Special:NewPagesFeed:22), <anonymous>:3:3373)
    at child.template (<anonymous>:22:874)
    at child.render (<anonymous>:197:1336)
    at child.addOne (<anonymous>:217:449)
    at Array.forEach (<anonymous>)
    at Function._.each._.forEach (<anonymous>:8:69)
    at child.Collection.(anonymous function) [as forEach] (<anonymous>:37:683)
    at child.addAll (<anonymous>:217:939)
    at child.trigger (<anonymous>:25:695)
    at child.reset (<anonymous>:35:440)

Details

	Subject	Repo	Branch	Lines +/-
	Check required metadata properties in list API and compilation job	mediawiki/extensions/PageTriage	master	+27 -3

Customize query in gerrit

Related Objects

Mentioned In: T221815: "Next article" button in toolbar is broken when feed is sorted by oldest articles
T179220: Buttons on the curation toolbar stay blue
Mentioned Here: rEPTR812a42f379a4: ArticleCompileInterface: Use Article::getParserOutput() directly

Event Timeline

Force_Radical created this task.Feb 21 2019, 4:31 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 21 2019, 4:31 AM

Force_Radical triaged this task as Unbreak Now! priority.Feb 21 2019, 4:33 AM

Force_Radical added a project: Growth-Team.

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptFeb 21 2019, 4:33 AM

Force_Radical added a project: PageTriage.Feb 21 2019, 5:24 AM

@SBisson @kostajh @Catrope -- I am able to reproduce this issue. I'm tagging you in case the problem and solution are immediately obvious to you. We can discuss this in an upcoming team meeting.

FYI to @Niharika, whose team will soon be working on the New Pages Feed.

DannyS712 subscribed.Feb 21 2019, 11:33 AM

This is caused by the API retuning incomplete data for the latest revision of Battle of Choirokoitia, which was made anonymously on Feb 20, 2019. At the time of writing this, it's the first result returned by this API query.

CASSIOPEIA subscribed.Feb 21 2019, 1:23 PM

kostajh mentioned this in T179220: Buttons on the curation toolbar stay blue.Feb 21 2019, 2:11 PM

I have fix this specific page's metadata by running the following on a maintenance server.

$acp = MediaWiki\Extension\PageTriage\ArticleCompile\ArticleCompileProcessor::newFromPageId( [ 4306709 ] );
$acp->compileMetadata();

That fixes Special:NewPagesFeed in the short term but there's certainly a bug that caused this metadata to be compiled incomplete in the first place.

Thanks, @SBisson. I'm going to put this in our team's "Revisit" column, so that we can keep an eye on how often it happens. Then we can decide whether to dig into the underlying issues. @Niharika's team may also discover this in their work.

Thanks for the quick fix, @SBisson.

I am currently experiencing this issue.

Thanks, @Barkeep49.

I am moving this back onto our sprint board so our team can discuss tomorrow. Hopefully we can do a quick fix to make the feed work again and then start on fixing the underlying issue.

MMiller_WMF moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.Feb 26 2019, 11:52 PM

The JSON which is breaking the feed, as returned by this query:

{
                "ores_draftquality": "",
                "ores_articlequality": "B-class",
                "creator_contribution_page_url": "//en.wikipedia.org/wiki/Special:Contributions/",
                "creator_contribution_page": "Special:Contributions/",
                "creator_user_talk_page_exist": false,
                "creator_user_talk_page_url": "//en.wikipedia.org/wiki/User_talk:",
                "creator_user_talk_page": "User talk:",
                "creator_user_page_exist": false,
                "creator_user_page_url": "//en.wikipedia.org/wiki/User:",
                "creator_user_page": "User:",
                "creation_date_utc": "20051117201903",
                "user_name": "",
                "user_id": "",
                "user_experience": "",
                "user_editcount": "",
                "user_creation_date": "",
                "user_bot": "",
                "user_block_status": "",
                "user_autoconfirmed": "",
                "snippet": "",
                "rev_count": "",
                "reference": "",
                "prod_status": "",
                "page_len": "",
                "linkcount": "",
                "csd_status": "",
                "category_count": "",
                "blp_prod_status": "",
                "afd_status": "",
                "afc_state": "",
                "title": "Merrill Lynch & Co.",
                "reviewer": null,
                "ptrp_reviewed_updated": "20190226224641",
                "ptrp_last_reviewed_by": "0",
                "is_redirect": "0",
                "patrol_status": "0",
                "creation_date": "20051117201903",
                "copyvio": "885254066",
                "pageid": "3185291"
            }

The data as it exists in the DB:

> $articles = MediaWiki\Extension\PageTriage\ArticleMetadata::getMetadataForArticles( [ 3185291 ] );

> print_r($articles);
Array
(
    [3185291] => Array
        (
            [copyvio] => 885254066
            [creation_date] => 20051117201903
            [patrol_status] => 0
            [is_redirect] => 0
            [ptrp_last_reviewed_by] => 0
            [ptrp_reviewed_updated] => 20190226224641
            [reviewer] =>
            [title] => Merrill Lynch & Co.
        )

)

What this and T216674#4971774 have in common is the removal of a redirect.

The feed is fixed now; the article in question had two edits which triggered metadata recompilation.

My suggestions for this are

add a more robust check here:

if ( !isset( $metaData[$page] ) ) {
  // If metadata is missing for a page, add this to the API output.
  $result['pages_missing_metadata'][] = $page;
  $result['result'] = 'warning';
  continue;
}

We could establish a list of the required metadata fields, without which the UI breaks. Below this block of code, we could have another statement to check that metadata exists and that required fields are populated, and if not, the page is reported as missing metadata. This time we could add some logging so we have a better idea of the frequency of the problem.

In looking at the logs, there's an error on metadata compilation, with the message Using cached lag value for {db_server} due to active transaction. Backtrace:

#0 /srv/mediawiki/php-1.33.0-wmf.18/includes/libs/rdbms/database/DatabaseMysqlBase.php(719): Wikimedia\Rdbms\DatabaseMysqlBase->getLagFromPtHeartbeat()
#1 /srv/mediawiki/php-1.33.0-wmf.18/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1841): Wikimedia\Rdbms\DatabaseMysqlBase->getLag()
#2 /srv/mediawiki/php-1.33.0-wmf.18/includes/specials/SpecialContributions.php(240): Wikimedia\Rdbms\LoadBalancer->safeGetLag()
#3 /srv/mediawiki/php-1.33.0-wmf.18/includes/specialpage/SpecialPage.php(569): SpecialContributions->execute()
#4 /srv/mediawiki/php-1.33.0-wmf.18/includes/specialpage/SpecialPageFactory.php(558): SpecialPage->run()
#5 /srv/mediawiki/php-1.33.0-wmf.18/includes/specialpage/SpecialPageFactory.php(616): MediaWiki\Special\SpecialPageFactory->executePath()
#6 /srv/mediawiki/php-1.33.0-wmf.18/includes/parser/Parser.php(3317): MediaWiki\Special\SpecialPageFactory->capturePath()
#7 /srv/mediawiki/php-1.33.0-wmf.18/includes/parser/Preprocessor_Hash.php(1114): Parser->braceSubstitution()
#8 /srv/mediawiki/php-1.33.0-wmf.18/includes/parser/Parser.php(3036): PPFrame_Hash->expand()
#9 /srv/mediawiki/php-1.33.0-wmf.18/includes/parser/Parser.php(1354): Parser->replaceVariables()
#10 /srv/mediawiki/php-1.33.0-wmf.18/includes/parser/Parser.php(482): Parser->internalParse()
#11 /srv/mediawiki/php-1.33.0-wmf.18/includes/content/WikitextContent.php(369): Parser->parse()
#12 /srv/mediawiki/php-1.33.0-wmf.18/includes/content/AbstractContent.php(555): WikitextContent->fillParserOutput()
#13 /srv/mediawiki/php-1.33.0-wmf.18/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput()
#14 /srv/mediawiki/php-1.33.0-wmf.18/includes/Revision/RenderedRevision.php(234): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#15 /srv/mediawiki/php-1.33.0-wmf.18/includes/Revision/RevisionRenderer.php(193): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#16 /srv/mediawiki/php-1.33.0-wmf.18/includes/Revision/RevisionRenderer.php(142): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#17 (): Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2()
#18 /srv/mediawiki/php-1.33.0-wmf.18/includes/Revision/RenderedRevision.php(197): call_user_func()
#19 /srv/mediawiki/php-1.33.0-wmf.18/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#20 /srv/mediawiki/php-1.33.0-wmf.18/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#21 /srv/mediawiki/php-1.33.0-wmf.18/includes/page/WikiPage.php(1244): PoolCounterWork->execute()
#22 /srv/mediawiki/php-1.33.0-wmf.18/extensions/PageTriage/includes/ArticleCompile/ArticleCompileInterface.php(126): WikiPage->getParserOutput()
#23 /srv/mediawiki/php-1.33.0-wmf.18/extensions/PageTriage/includes/ArticleCompile/ArticleCompileCategoryCount.php(16): MediaWiki\Extension\PageTriage\ArticleCompile\ArticleCompileInterface->getParserOutputByPageId()
#24 /srv/mediawiki/php-1.33.0-wmf.18/extensions/PageTriage/includes/ArticleCompile/ArticleCompileProcessor.php(259): MediaWiki\Extension\PageTriage\ArticleCompile\ArticleCompileCategoryCount->compile()
#25 /srv/mediawiki/php-1.33.0-wmf.18/extensions/PageTriage/includes/ArticleCompile/ArticleCompileProcessor.php(190): MediaWiki\Extension\PageTriage\ArticleCompile\ArticleCompileProcessor->process()
#26 /srv/mediawiki/php-1.33.0-wmf.18/extensions/PageTriage/includes/Hooks.php(201): MediaWiki\Extension\PageTriage\ArticleCompile\ArticleCompileProcessor->compileMetadata()
#27 /srv/mediawiki/php-1.33.0-wmf.18/includes/deferred/MWCallableUpdate.php(34): Closure$MediaWiki\Extension\PageTriage\Hooks::onLinksUpdateComplete()
#28 /srv/mediawiki/php-1.33.0-wmf.18/includes/deferred/DeferredUpdates.php(270): MWCallableUpdate->doUpdate()
#29 /srv/mediawiki/php-1.33.0-wmf.18/includes/deferred/DeferredUpdates.php(228): DeferredUpdates::runUpdate()
#30 /srv/mediawiki/php-1.33.0-wmf.18/includes/deferred/DeferredUpdates.php(140): DeferredUpdates::execute()
#31 /srv/mediawiki/php-1.33.0-wmf.18/extensions/EventBus/includes/JobExecutor.php(90): DeferredUpdates::doUpdates()
#32 /srv/mediawiki/rpc/RunSingleJob.php(77): JobExecutor->execute()
#33 {main}

This code is triggered by a change made in September rEPTR812a42f379a4: ArticleCompileInterface: Use Article::getParserOutput() directly. We should investigate this further as a possible cause of the metadata compilation failure.

kostajh claimed this task.Feb 28 2019, 3:27 PM

kostajh changed the subtype of this task from "Task" to "Bug Report".

kostajh moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.

Change 493505 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/PageTriage@master] Check required metadata properties in list API and compilation job

https://gerrit.wikimedia.org/r/493505

gerritbot added a project: Patch-For-Review.Feb 28 2019, 8:48 PM

kostajh moved this task from In Progress to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.Feb 28 2019, 10:01 PM

I've +2 the patch above. AFAICT it is not trying to fix the problem but only prevent it from breaking the page and add some logs.

Should it go back to "in progress" for the root issue?

Change 493505 merged by jenkins-bot:
[mediawiki/extensions/PageTriage@master] Check required metadata properties in list API and compilation job

https://gerrit.wikimedia.org/r/493505

@SBisson -- in case that's a question for me, I just want to say that I think that's a better question for @kostajh. I don't have a preference; it depends on the consequences of not fixing the root issue. What are they?

AFAICT it is not trying to fix the problem but only prevent it from breaking the page and add some logs.
Should it go back to "in progress" for the root issue?

I would like to see what we learn from the logs about the frequency of the problem. I didn't see any obvious solutions when I worked through the backtrace. If the patch successfully prevents the page from breaking then IMO this task could be moved to a queue of tasks that we or someone else can work on when there's time, but not necessarily in an upcoming sprint.

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.21; 2019-03-12).Mar 5 2019, 4:01 PM

Moving to QA per the above.

The patch fixed the problem with that page revision and it was deployed to wmf.20. I looked into logstash - no recent logs for the problem.
Only one WARNING about missing metadata:

Article metadata not found in DB, will attempt to save to DB via the job queue.

{
  "_index": "logstash-2019.03.09",
  "_type": "mediawiki",
  "_id": "AWlgJZGc8aQffZ3HH9xi",
  "_version": 1,
  "_score": null,
  "_source": {
    "server": "en.wikipedia.org",
    "phpversion": "7.2.8-1+0~20180725124257.2+stretch~1.gbp571e56",
    "wiki": "enwiki",
    "channel": "PageTriage",
    "program": "mediawiki",
    "type": "mediawiki",
    "trace": "#0 /srv/mediawiki/php-1.33.0-wmf.20/extensions/PageTriage/includes/ArticleMetadata.php(212): MediaWiki\\Extension\\PageTriage\\ArticleCompile\\ArticleCompileProcessor->compileMetadata(2)\n#1 /srv/mediawiki/php-1.33.0-wmf.20/extensions/PageTriage/includes/Api/ApiPageTriageList.php(86): MediaWiki\\Extension\\PageTriage\\ArticleMetadata->getMetadata()\n#2 /srv/mediawiki/php-1.33.0-wmf.20/includes/api/ApiMain.php(1595): MediaWiki\\Extension\\PageTriage\\Api\\ApiPageTriageList->execute()\n#3 /srv/mediawiki/php-1.33.0-wmf.20/includes/api/ApiMain.php(531): ApiMain->executeAction()\n#4 /srv/mediawiki/php-1.33.0-wmf.20/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()\n#5 /srv/mediawiki/php-1.33.0-wmf.20/api.php(87): ApiMain->execute()\n#6 /srv/mediawiki/w/api.php(3): require('/srv/mediawiki/...')\n#7 {main}",
    "http_method": "GET",
    "host": "mw1289",
    "@version": 1,
    "shard": "s1",
    "timestamp": "2019-03-09T01:51:08+00:00",
    "severity": "warning",
    "unique_id": "XIMcDApAEDYAAHQS1igAAACI",
    "level": "WARNING",
    "ip": "10.64.48.103",
    "mwversion": "1.33.0-wmf.20",
    "logsource": "mw1289",
    "message": "Article metadata not found in DB, will attempt to save to DB via the job queue.",
    "normalized_message": "Article metadata not found in DB, will attempt to save to DB via the job queue.",
    "url": "/w/api.php?action=pagetriagelist&format=json&page_id=60177002",
    "reqId": "XIMcDApAEDYAAHQS1igAAACI",
    "tags": [
      "input-kafka-rsyslog-udp-localhost",
      "rsyslog-udp-localhost",
      "kafka",
      "es"
    ],
    "referrer": "https://en.wikipedia.org/wiki/Sons_of_Antiochus_VIII",
    "@timestamp": "2019-03-09T01:51:08.556Z",
    "raw_query_string": "action=pagetriagelist&format=json&page_id=60177002",
    "facility": "user",
    "articles_without_metadata": "60177002"
  },
  "fields": {
    "@timestamp": [
      1552096268556
    ]
  },
  "sort": [
    1552096268556
  ]
}

Liuxinyu970226 unsubscribed.Mar 11 2019, 12:27 AM

DannyS712 mentioned this in T221815: "Next article" button in toolbar is broken when feed is sorted by oldest articles.Apr 25 2019, 5:22 AM

NewPagesFeed not working when sorted oldest firstClosed, ResolvedPublicBUG REPORTActions