Page MenuHomePhabricator
Create Task
Maniphest T338292

Add sentence segmenter feature
Closed, ResolvedPublic
Actions

Description

As the NMT models work on sentences, translating all sentences in a paragraph in parallel gives better performance. The APIs of translation models are defined in this way. CXServer currently does this segmentation before sending the content to MinT. But, for the usecase where cxserver is not needed, the segmentation should happen inside the MinT api, so that consumers of translation api can sent Paragraphs directly.

Details

SubjectRepoBranchLines +/-
Update MinT to 2023-08-02-142037-productionoperations/deployment-chartsmaster+1 -1
Sentence segmentermediawiki/services/machinetranslationmaster+505 -37
Customize query in gerrit

Related Objects

Event Timeline

santhosh created this task.Jun 7 2023, 5:43 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2023, 5:43 AM
santhosh moved this task from Backlog to Infrastructure on the MinT board.Jun 7 2023, 5:43 AM
santhosh moved this task from Infrastructure to General translation functionality on the MinT board.
Pginer-WMF triaged this task as Medium priority.Jul 27 2023, 10:54 AM
Pginer-WMF added a project: Language-Team (Language-2023-July-September).
gerritbot added a comment.Jul 31 2023, 10:58 AM

Change 929682 had a related patch set uploaded (by Santhosh; author: Santhosh):

[mediawiki/services/machinetranslation@master] Sentence segmenter

https://gerrit.wikimedia.org/r/929682

gerritbot added a project: Patch-For-Review.Jul 31 2023, 10:58 AM
Pginer-WMF moved this task from Quarter Backlog to Priority: Translation on the Language-Team (Language-2023-July-September) board.Aug 1 2023, 11:40 AM
gerritbot added a comment.Aug 2 2023, 7:59 AM

Change 929682 merged by jenkins-bot:

[mediawiki/services/machinetranslation@master] Sentence segmenter

https://gerrit.wikimedia.org/r/929682

Maintenance_bot removed a project: Patch-For-Review.Aug 2 2023, 8:10 AM
gerritbot added a comment.Aug 3 2023, 4:48 AM

Change 945036 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update MinT to 2023-08-02-142037-production

https://gerrit.wikimedia.org/r/945036

gerritbot added a project: Patch-For-Review.Aug 3 2023, 4:48 AM
gerritbot added a comment.Aug 3 2023, 6:01 AM

Change 945036 merged by jenkins-bot:

[operations/deployment-charts@master] Update MinT to 2023-08-02-142037-production

https://gerrit.wikimedia.org/r/945036

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2023, 6:10 AM
Stashbot added a comment.Aug 3 2023, 6:31 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-03T06:31:50Z] <kart_> Updated MinT to 2023-08-02-142037-production (T338292)

santhosh claimed this task.Aug 8 2023, 7:51 AM
santhosh mentioned this in T343781: Expand sentence segmentation system.Aug 8 2023, 7:58 AM
Pginer-WMF closed this task as Resolved.Sep 26 2023, 10:59 AM
Pginer-WMF moved this task from Priority: Translation to Done on the Language-Team (Language-2023-July-September) board.
Pginer-WMF mentioned this in T347389: Integrate improved sentence segmentation algorithm in CXServer.Sep 26 2023, 11:08 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL