Page MenuHomePhabricator
Create Task
Maniphest T240550

Add mapping for ORES topic field in ElasticSearch
Closed, ResolvedPublic
Actions

Description

For making articles searchable via topic, ElasticSearch needs to index them via ORES drafttopic scores. ORES predictions come in the form of a topic name -> probability map (example); the best tool to implement them in ElasticSearch seems to be a text field, with topics stored as words and scores as word frequencies (meaning probabilities need to be scaled and rounded to some integer range; 0-1000 should be accurate enough).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Enable ORES articletopic handling in cirrussearch rolemediawiki/vagrantmaster+6 -0
Enable ores_articletopics field for all wikisoperations/mediawiki-configmaster+9 -0
Add ores_drafttopics field mappingmediawiki/extensions/CirrusSearchmaster+342 -1
Add ores_drafttopics field mappingmediawiki/extensions/GrowthExperimentsmaster+312 -0
[cirrus] add elastic mapping for ores drafttopicsoperations/mediawiki-configmaster+42 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Tgr created this task.Dec 12 2019, 10:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 12 2019, 10:57 AM
Tgr added a parent task: T240517: [EPIC] Growth: Newcomer tasks 1.1.1 (ORES topics).Dec 12 2019, 10:57 AM
Tgr mentioned this in T240517: [EPIC] Growth: Newcomer tasks 1.1.1 (ORES topics).Dec 12 2019, 11:08 AM
Tgr added a parent task: T240558: Update ORES articletopic data score in ElasticSearch when an article gets edited.Dec 12 2019, 11:23 AM
EBernhardson moved this task from needs triage to Current work on the Discovery-Search board.Dec 12 2019, 8:33 PM
EBernhardson edited projects, added Discovery-Search (Current work); removed Discovery-Search.
gerritbot added a comment.Dec 17 2019, 4:17 PM

Change 558577 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/mediawiki-config@master] [WIP] [cirrus] add elastic mapping for ores drafttopics

https://gerrit.wikimedia.org/r/558577

gerritbot added a project: Patch-For-Review.Dec 17 2019, 4:17 PM
dcausse moved this task from Incoming to Needs review on the Discovery-Search (Current work) board.Dec 17 2019, 6:35 PM
EBernhardson assigned this task to dcausse.Dec 17 2019, 11:50 PM
EBernhardson triaged this task as Medium priority.
MMiller_WMF removed a parent task: T240517: [EPIC] Growth: Newcomer tasks 1.1.1 (ORES topics).Dec 18 2019, 1:52 AM
gerritbot added a comment.Dec 18 2019, 5:18 PM

Change 559142 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/GrowthExperiments@master] Add ores_drafttopics field mapping

https://gerrit.wikimedia.org/r/559142

gerritbot added a comment.Dec 18 2019, 6:52 PM

Change 558577 abandoned by DCausse:
[cirrus] add elastic mapping for ores drafttopics

https://gerrit.wikimedia.org/r/558577

gerritbot added a comment.Dec 19 2019, 4:20 PM

Change 559142 abandoned by DCausse:
Add ores_drafttopics field mapping

Reason:
will move this inside CirrusSearch directly

https://gerrit.wikimedia.org/r/559142

Maintenance_bot removed a project: Patch-For-Review.Dec 19 2019, 5:10 PM
gerritbot added a comment.Dec 20 2019, 5:13 PM

Change 559911 had a related patch set uploaded (by DCausse; owner: DCausse):
[mediawiki/extensions/CirrusSearch@master] Add ores_drafttopics field mapping

https://gerrit.wikimedia.org/r/559911

gerritbot added a project: Patch-For-Review.Dec 20 2019, 5:13 PM
gerritbot added a comment.Jan 8 2020, 1:44 AM

Change 559911 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Add ores_drafttopics field mapping

https://gerrit.wikimedia.org/r/559911

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.15; 2020-01-14).Jan 8 2020, 2:00 AM
Maintenance_bot removed a project: Patch-For-Review.Jan 8 2020, 2:10 AM
MMiller_WMF subscribed.Jan 22 2020, 5:43 AM
EBernhardson mentioned this in T147505: [tracking] CirrusSearch: what is updated during re-indexing.Feb 10 2020, 6:40 PM
Tgr added a comment.Feb 13 2020, 12:19 AM

@dcausse this task has been resolved, right?

EBernhardson subscribed.Feb 18 2020, 9:48 PM

This is still waiting for an in-place reindex before it is queryable. We were waiting on wmf.19 and an unrelated mapping change before running that. Now that that change is deployed along with this one we should be able to run the re-index this week.

Tgr mentioned this in T240556: Load ORES articletopic data into ElasticSearch via the weekly bulk update.Feb 18 2020, 10:21 PM
gerritbot added a comment.Feb 18 2020, 10:38 PM

Change 573003 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[operations/mediawiki-config@master] Enable ores_articletopics field for all wikis

https://gerrit.wikimedia.org/r/573003

gerritbot added a project: Patch-For-Review.Feb 18 2020, 10:38 PM
gerritbot added a comment.Feb 18 2020, 10:51 PM

Change 573003 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable ores_articletopics field for all wikis

https://gerrit.wikimedia.org/r/573003

Maintenance_bot removed a project: Patch-For-Review.Feb 18 2020, 11:11 PM
gerritbot added a comment.Feb 19 2020, 2:16 AM

Change 573056 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[mediawiki/vagrant@master] Enable ORES articletopic handling in cirrussearch role

https://gerrit.wikimedia.org/r/573056

gerritbot added a project: Patch-For-Review.Feb 19 2020, 2:16 AM
gerritbot added a comment.Feb 19 2020, 9:27 AM

Change 573056 merged by jenkins-bot:
[mediawiki/vagrant@master] Enable ORES articletopic handling in cirrussearch role

https://gerrit.wikimedia.org/r/573056

Maintenance_bot removed a project: Patch-For-Review.Feb 19 2020, 10:11 AM
EBernhardson added a comment.Feb 20 2020, 5:19 PM

This reindex process is running, will probably finish late next week

Tgr added a comment.Feb 20 2020, 6:13 PM

Thanks @EBernhardson!

Does that block T243357: Once the ORES articletopic - ElasticSearch pipeline is set up, update data about all articles? I figured we could do it now so we don't have to wait a week for the full dataset.

EBernhardson added a comment.Feb 20 2020, 6:14 PM

The reindex wont block anything, essentially elasticsearch will store all the data we send to it, but it's only searchable once the reindex process makes it to that wiki.

EBernhardson moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.Feb 27 2020, 11:15 PM
EBernhardson moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.
EBernhardson moved this task from Needs Reporting to To Be Deployed on the Discovery-Search (Current work) board.
dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Mar 2 2020, 6:38 PM
dcausse moved this task from Needs Reporting to To Be Deployed on the Discovery-Search (Current work) board.
Tgr added a comment.Apr 2 2020, 2:45 PM

Search is working now on zhwiki so probably finished?

dcausse added a subscriber: Mstyles.Apr 2 2020, 2:48 PM

@Tgr yes the reindex is done but beware that it's quite common for some wikis to fail, @Mstyles will double check the ones that failed and rerun the reindex on them.

dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Apr 20 2020, 5:30 PM
TJones closed this task as Resolved.Apr 29 2020, 2:37 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits