Page MenuHomePhabricator
Create Task
Maniphest T89280

Add Japanese segmentation to cxserver
Closed, ResolvedPublic
Actions

Description

Japanese uses non-Western sentence separators, so it needs its own segmentation algorithm.

AFAIK, its needs are almost the same as those of Hindi, but instead of Danda ("।") it needs an ideographic full stop ("。"). This is what I made in https://gerrit.wikimedia.org/r/#/c/189984/ .

Niklas also suggests to consider "「", "」" and "．". AFAIK, "「" and "」" are like quotation marks and not sentence separators. As for "．", I'll need consultation from somebody who knows Japanese; my hunch is that it's not needed.

Details

SubjectRepoBranchLines +/-
Add Japanese segmentationmediawiki/services/cxservermaster+43 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Amire80 created this task.Feb 11 2015, 7:49 PM
Amire80 raised the priority of this task from to Medium.
Amire80 updated the task description. (Show Details)
Amire80 added projects: ContentTranslation, ContentTranslation-CXserver, Patch-For-Review.
Amire80 added subscribers: Amire80, dchan, • Jsahleen, santhosh.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 11 2015, 7:49 PM
Amire80 added a parent task: T89240: enable ContentTranslation in Japanese on testing servers.Feb 11 2015, 7:50 PM
gerritbot added a subscriber: gerritbot.Feb 11 2015, 7:51 PM

Change 189984 had a related patch set uploaded (by Amire80):
Add Japanese segmentation

https://gerrit.wikimedia.org/r/189984

Patch-For-Review

gerritbot added a comment.Feb 12 2015, 10:16 AM

Change 189984 merged by jenkins-bot:
Add Japanese segmentation

https://gerrit.wikimedia.org/r/189984

Liuxinyu970226 added a subscriber: Liuxinyu970226.Feb 12 2015, 10:44 AM
Asahiko added a subscriber: Asahiko.Mar 18 2015, 7:06 PM
Asahiko added a comment.Mar 18 2015, 7:36 PM

A comment from a native Japanese speaker: Generally, the ideographic full stop "。" is the equivalent of a latin-script full stop and is the most common sentence separator. (Same goes for Chinese.)

"「" and "」" are indeed the equivalents of quotation marks.

The fullwidth full stop "．" is also a valid way to end a sentence in horizontal writing mode for Japanese, but is less common except in certain technical documents. The distinction is a manual-of-style matter. Japanese Wikipedia uses the ideographic full stop "。", as is the case with any common usage. Although one cannot rule out the possibility of encountering the fullwidth form, I would say that it would be extremely rare.

Amire80 mentioned this in rGCXS21b0028adcc4: Add Japanese segmentation.Apr 8 2015, 4:12 PM
Amire80 closed this task as Resolved.May 16 2015, 1:15 PM
Amire80 claimed this task.

Thanks for the comments, @Asahiko.

This is actually done long ago and I forgot to close it :)

Amire80 removed a project: Patch-For-Review.May 16 2015, 1:16 PM
Amire80 set Security to None.
Liuxinyu970226 removed a subscriber: Liuxinyu970226.May 16 2015, 1:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL