Page MenuHomePhabricator
Create Task
Maniphest T111905

Design the technical infrastructure for parallel corpora storage and api (tracking)
Closed, ResolvedPublic
Actions

Description

Information about published translations are generally helpful for machine translation developers and others for different purposes. Content translation aims to provide this data as much as possible in different ways. The amount and details of the data will be improved over time.

The current state is documented at https://www.mediawiki.org/wiki/Content_translation/Published_translations

As next step we need to design and implement a parallel corpora serving infrastructure.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedsanthoshT95886 for ContentTranslation MT, store information about source content, machine-translated content and user-edited content
 ResolvedsanthoshT111905 Design the technical infrastructure for parallel corpora storage and api (tracking)
 ResolvedsanthoshT119489 Avoid deleting the saved translation once translation published to wiki
 ResolvedsanthoshT119495 Save the revision id of source and published article in cx_translations table
 ResolvedjcrespoT119752 Review schema changes for T119495
 ResolvedjcrespoT121509 Correct the schema to make revision id columns of cx_translations nullable
 ResolvedsanthoshT119497 Expose the revisions in ApiQueryPublishedTranslations output
ResolvedsanthoshT119617 Create new table for CX Parallel corpora
 ResolvedNoneT120815 Review and create CX Parallel corpora table
 Resolved• NikerabbitT119618 Create API for accessing parallel corpora by providing translation id
 ResolvedsanthoshT120062 Design and develop API to save the source, initial MT and final translation at section level to parallel corpora table
 Resolved• NikerabbitT122042 Provide parallel corpora dumps
 ResolvedsanthoshT124399 Migrate the draft translation restore mechanism to use data from cx_corpora table
 ResolvedsanthoshT125621 ApiQueryContentTranslation: Serve the saved translation from cxc_corpora table
 ResolvedsanthoshT125622 While saving the data to corpora table, remove the corresponding entry from drafts table
 ResolvedsanthoshT125623 Remove the ext.cx.drafts module, migrate the section restore logic to a new module
 ResolvedsanthoshT125645 Use cxpublish api only for publish and use cxsave api for save
 ResolvedsanthoshT128162 Resuming a re-translated deleted translation will cause restoring the sections from old deleted translation
 ResolvedsanthoshT130178 Translation restore: Do not attempt adding orphan section unless source article is old revision
 ResolvedArielGlennT127793 Create Content Translation Parallel Corpora dumps
 Resolved• NikerabbitT133006 Add compression support to scripts/dump-corpora.php
 Resolved• NikerabbitT133007 Add --output option to scripts/dump-corpora.php

Event Timeline

santhosh created this task.Sep 9 2015, 7:33 AM
santhosh claimed this task.
santhosh raised the priority of this task from to Medium.
santhosh updated the task description. (Show Details)
santhosh added projects: OKR-Work, ContentTranslation-Release6, ContentTranslation, LE-CX6-Sprint 4.
santhosh added subscribers: Unhammer, He7d3r, KuboF and 8 others.
Amire80 moved this task from Needs Triage to CX7 on the ContentTranslation board.Sep 9 2015, 8:35 AM
Amire80 moved this task from Backlog to Parallel corpora on the ContentTranslation-Release6 board.Sep 13 2015, 6:42 AM
Arrbee added a project: LE-CX6-Sprint break.Sep 23 2015, 7:30 AM
Arrbee set Security to None.
Arrbee removed a project: LE-CX6-Sprint 4.Sep 23 2015, 11:26 AM
santhosh edited projects, added LE-CX7-Sprint 1, ContentTranslation-Release7; removed LE-CX6-Sprint break, ContentTranslation-Release6.Oct 6 2015, 6:43 AM
Pginer-WMF moved this task from Backlog to 3rd party support on the ContentTranslation-Release7 board.Oct 6 2015, 7:28 AM
Amire80 removed a project: LE-CX7-Sprint 1.Oct 21 2015, 7:21 AM
Arrbee added a project: LE-CX7-Sprint 3.Nov 4 2015, 7:16 AM
Ryuch subscribed.Nov 11 2015, 7:11 AM
santhosh updated the task description. (Show Details)Nov 20 2015, 7:17 AM
Equ subscribed.Nov 20 2015, 8:35 AM
santhosh added subtasks: T119489: Avoid deleting the saved translation once translation published to wiki, T119495: Save the revision id of source and published article in cx_translations table, T119497: Expose the revisions in ApiQueryPublishedTranslations output .Nov 24 2015, 12:04 PM

The design being drafted at https://docs.google.com/document/d/11fFVBcu190u8J4uVJdyrghuHx5BdWrRHRZvd7fArwE0

Pginer-WMF subscribed.Nov 24 2015, 12:14 PM
santhosh moved this task from Backlog to In Progress on the LE-CX7-Sprint 3 board.Nov 25 2015, 7:01 AM
santhosh added a subtask: T119617: Create new table for CX Parallel corpora.Nov 25 2015, 12:10 PM
santhosh added a subtask: T119618: Create API for accessing parallel corpora by providing translation id.Nov 25 2015, 12:13 PM
Arrbee renamed this task from Design the technical infrastructure for parallel corpora storage and api to [Tracker] Design the technical infrastructure for parallel corpora storage and api.Dec 1 2015, 7:26 AM
Arrbee edited projects, added LE-CX7-Sprint 4; removed LE-CX7-Sprint 3.
KartikMistry closed subtask T119489: Avoid deleting the saved translation once translation published to wiki as Resolved.Dec 2 2015, 6:48 AM
santhosh raised the priority of this task from Medium to High.Dec 2 2015, 7:07 AM
santhosh moved this task from Backlog to In Progress on the LE-CX7-Sprint 4 board.Dec 10 2015, 11:50 AM
santhosh closed subtask T119497: Expose the revisions in ApiQueryPublishedTranslations output as Resolved.Dec 17 2015, 4:42 AM
santhosh added a subtask: T122042: Provide parallel corpora dumps.Dec 21 2015, 11:00 AM
santhosh edited projects, added LE-CX7-Sprint break; removed LE-CX7-Sprint 4.Dec 22 2015, 6:44 AM
santhosh closed subtask T119495: Save the revision id of source and published article in cx_translations table as Resolved.Dec 22 2015, 7:03 AM
jmadler subscribed.Jan 6 2016, 5:15 AM
santhosh edited projects, added ContentTranslation-Release8, LE-CX8-Sprint 1; removed LE-CX7-Sprint break, ContentTranslation-Release7.Jan 18 2016, 6:25 AM
santhosh moved this task from Backlog to In Progress on the LE-CX8-Sprint 1 board.Jan 18 2016, 6:58 AM
Amire80 moved this task from Backlog to High-priority Bugs on the ContentTranslation-Release8 board.Jan 20 2016, 3:29 PM
Amire80 moved this task from High-priority Bugs to Parallel corpora on the ContentTranslation-Release8 board.
KartikMistry closed subtask T119617: Create new table for CX Parallel corpora as Resolved.Jan 22 2016, 4:26 AM
santhosh added a comment.Jan 22 2016, 6:43 AM

Documentation: https://www.mediawiki.org/wiki/Content_translation/Published_translations#Parallel_corpora

santhosh closed subtask T119618: Create API for accessing parallel corpora by providing translation id as Resolved.Jan 22 2016, 10:21 AM
santhosh closed subtask T120062: Design and develop API to save the source, initial MT and final translation at section level to parallel corpora table as Resolved.Jan 22 2016, 10:25 AM
Amire80 moved this task from CX7 to CX8 on the ContentTranslation board.Jan 24 2016, 10:30 PM
Amire80 moved this task from CX8 to CX7 on the ContentTranslation board.
Arrbee edited projects, added LE-CX8-Sprint 2; removed LE-CX8-Sprint 1.Feb 2 2016, 6:36 AM
santhosh moved this task from Backlog to In Progress on the LE-CX8-Sprint 2 board.Feb 3 2016, 4:18 AM
KuboF unsubscribed.Feb 6 2016, 4:50 PM
Amire80 moved this task from CX7 to CX8 on the ContentTranslation board.Feb 14 2016, 4:32 PM
Arrbee edited projects, added LE-CX8-Sprint 3; removed LE-CX8-Sprint 2.Feb 16 2016, 6:51 AM
santhosh moved this task from Backlog to In Review on the LE-CX8-Sprint 3 board.Feb 16 2016, 6:57 AM
santhosh moved this task from In Review to In Progress on the LE-CX8-Sprint 3 board.
Arrbee added a subtask: T127793: Create Content Translation Parallel Corpora dumps.Mar 4 2016, 6:59 AM
Arrbee closed subtask T122042: Provide parallel corpora dumps as Resolved.
Arrbee closed subtask T124399: Migrate the draft translation restore mechanism to use data from cx_corpora table as Resolved.Mar 4 2016, 7:37 AM
Arrbee removed a project: LE-CX8-Sprint 3.Mar 4 2016, 7:59 AM
santhosh mentioned this in T95886: for ContentTranslation MT, store information about source content, machine-translated content and user-edited content.Mar 7 2016, 6:47 AM
Arrbee edited projects, added ContentTranslation-Release9; removed ContentTranslation-Release8.Apr 20 2016, 9:42 AM
Restricted Application added a project: Language-Engineering April-June 2016. · View Herald TranscriptApr 20 2016, 9:42 AM
Arrbee moved this task from Backlog to Parallel corpora on the ContentTranslation-Release9 board.Apr 20 2016, 10:02 AM
Amire80 moved this task from Backlog to CX9 on the Language-Engineering April-June 2016 board.Apr 20 2016, 10:17 AM
Amire80 moved this task from CX8 to CX9 on the ContentTranslation board.Apr 20 2016, 12:32 PM
Danny_B renamed this task from [Tracker] Design the technical infrastructure for parallel corpora storage and api to Design the technical infrastructure for parallel corpora storage and api (tracking).May 31 2016, 3:22 PM
Danny_B added a project: Tracking-Neverending.
akosiaris changed the status of subtask T127793: Create Content Translation Parallel Corpora dumps from Open to Stalled.Jun 7 2016, 8:39 AM
Amire80 moved this task from CX9 to CX10 (July-Sep. 2016) on the ContentTranslation board.Jul 20 2016, 6:21 AM
Arrbee edited projects, added Language-Engineering July-September 2016; removed Language-Engineering April-June 2016.Jul 21 2016, 8:36 AM
Restricted Application added a project: Language-Engineering April-June 2016. · View Herald TranscriptJul 21 2016, 8:36 AM
Arrbee removed a project: Language-Engineering April-June 2016.Jul 21 2016, 8:36 AM
Restricted Application added a project: Language-Engineering April-June 2016. · View Herald TranscriptJul 21 2016, 8:36 AM
Arrbee removed projects: Language-Engineering April-June 2016, ContentTranslation-Release9.Jul 21 2016, 8:39 AM
Arrbee added a project: ContentTranslation-Release10.
• Nikerabbit changed the status of subtask T127793: Create Content Translation Parallel Corpora dumps from Stalled to Open.Jul 28 2016, 7:16 AM
Amire80 moved this task from Backlog to Parallel corpora on the ContentTranslation-Release10 board.Jul 31 2016, 11:21 AM
Amire80 moved this task from Backlog to CX10 on the Language-Engineering July-September 2016 board.Jul 31 2016, 11:27 AM
ArielGlenn closed subtask T127793: Create Content Translation Parallel Corpora dumps as Resolved.Aug 9 2016, 3:39 PM
Amire80 added a comment.Sep 7 2016, 12:35 PM

Can this be marked as done?

Amire80 closed this task as Resolved.Oct 3 2016, 11:14 AM
Amire80 mentioned this in T147185: Write a blog post about CX parallel corpora.
Amire80 added a project: Epic.

Marking as done.

There will definitely be future corpora work, but it can go in their own tasks.

This was an Epic. Thanks to everybody for getting this done!

Liuxinyu970226 unsubscribed.Oct 3 2016, 2:29 PM
Liuxinyu970226 moved this task from Tag to Transition completed / Archived on the Tracking-Neverending board.Nov 23 2018, 12:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits