Page MenuHomePhabricator
Create Task
Maniphest T206874

Add Nori (Korean) configuration to AnalysisConfigBuilder
Closed, ResolvedPublic
Actions

Description

The Nori Korean analyzer looks good to go, but since it is only available with ES 6.4.2+, we can't build it out yet.

  • Implement the configs in AnalysisConfigBuilder and add tests.
    • The command line version of the Elasticsearch config I used for testing so far is on MediaWiki.
  • Determine whether we need to change the config for plain field and the completion suggester.

For future reference: when it comes time to re-index, we will need to figure out how re-indexing Korean with a very different analyzer interacts with LTR.

Details

SubjectRepoBranchLines +/-
Configure Nori Korean analyzer for ES6mediawiki/extensions/CirrusSearches6+136 -5
[WIP] Add nori korean analyzeroperations/software/elasticsearch/pluginsmaster+13 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

TJones triaged this task as Medium priority.Oct 12 2018, 6:52 PM
TJones created this task.
Restricted Application added a subscriber: revi. · View Herald TranscriptOct 12 2018, 6:52 PM
ryucheol subscribed.Oct 25 2018, 11:03 AM
Ryuch subscribed.Oct 25 2018, 11:04 AM
gerritbot subscribed.Jan 24 2019, 11:51 AM

Change 486266 had a related patch set uploaded (by DCausse; owner: DCausse):
[operations/software/elasticsearch/plugins@master] [WIP] Add nori korean analyzer

https://gerrit.wikimedia.org/r/486266

gerritbot added a project: Patch-For-Review.Jan 24 2019, 11:51 AM
dcausse subscribed.Jan 24 2019, 11:52 AM

a test package with the korean analyzer is available here: https://people.wikimedia.org/~dcausse/wmf-elasticsearch-search-plugins_6.5.4-alpha2~stretch_all.deb

dcausse removed a project: Patch-For-Review.Jan 24 2019, 11:52 AM
TJones edited projects, added Discovery-Search; removed Discovery-Search (Current work).Jan 29 2019, 9:51 PM
TJones moved this task from needs triage to Language Stuff on the Discovery-Search board.
TJones moved this task from Language Stuff to Current work on the Discovery-Search board.Feb 13 2019, 7:00 PM
TJones edited projects, added Discovery-Search (Current work); removed Discovery-Search.
gerritbot added a comment.Feb 18 2019, 7:55 PM

Change 486266 abandoned by DCausse:
[WIP] Add nori korean analyzer

Reason:
test build only

https://gerrit.wikimedia.org/r/486266

gerritbot added a comment.Feb 20 2019, 3:51 PM

Change 491786 had a related patch set uploaded (by Tjones; owner: Tjones):
[mediawiki/extensions/CirrusSearch@es6] Configure Nori Korean analyzer for ES6

https://gerrit.wikimedia.org/r/491786

gerritbot added a project: Patch-For-Review.Feb 20 2019, 3:51 PM
gerritbot added a comment.Feb 21 2019, 10:50 AM

Change 491786 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@es6] Configure Nori Korean analyzer for ES6

https://gerrit.wikimedia.org/r/491786

TJones moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Feb 21 2019, 4:47 PM

We need to reindex, but not until after the ES6 upgrade is complete, and LTR has been disabled.

debt closed this task as Resolved.Feb 22 2019, 8:34 PM
debt subscribed.

Thanks for adding this to the ongoing index ticket (T147505), @TJones !

TJones closed subtask T216738: Reindex Korean-language wikis to enable Nori analyzer as Resolved.Apr 10 2019, 3:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL