Page MenuHomePhabricator
Create Task
Maniphest T164251

Use API action to preprocess text
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Instead of preprocessing text (running cleaner and segmenter) in onParserAfterTidy(), this should be done by the API action (which will be created in T164250). The response from the API (probably a JSON object) will be stored and used instead of the HTML elements (<utterances> and children) in the current implementation.

Since the text preprocessing will be run after the page is loaded, it should be possible to not process the whole page in one go. Instead, maybe process one paragraph at a time.

Details

ProjectBranchLines +/-Subject
mediawiki/extensions/Wikispeechmaster+1 K -1 KUse API to retrieve utterances
Customize query in gerrit

Related Objects
Search...

Event Timeline

Sebastian_Berlin-WMSE created this task.May 2 2017, 8:34 AM
Sebastian_Berlin-WMSE added a subtask: T164250: Create API action for getting segmented text.
Lokal_Profil moved this task from Incoming to Proposed for next sprint on the Wikispeech board.May 9 2017, 2:20 PM
Lokal_Profil set the point value for this task to 20.

In the first implementation have (js) trigger pre-processing on page load. As a follow-up task make this behaviour configurable (pre-process on play, user setting, url parameter etc.)

Lokal_Profil moved this task from Proposed for next sprint to Sprint 2017-05-10 on the Wikispeech board.May 9 2017, 3:03 PM
Lokal_Profil edited projects, added Wikispeech (Sprint 2017-05-10); removed Wikispeech.
Jopparn moved this task from Backlog to This Week on the User-Sebastian_Berlin-WMSE board.May 15 2017, 7:56 AM
Sebastian_Berlin-WMSE mentioned this in T165456: Sebastian at Wikimedia Hackathon 2017.May 16 2017, 7:07 AM
Sebastian_Berlin-WMSE added a comment.May 21 2017, 9:28 AM

While the API should accept parameters for cleaning and segmenting, these should still be kept in the config, both to be read by JS to make the request and for T164252: Keep functionality for preprocessing without javascript.

Sebastian_Berlin-WMSE moved this task from Sprint 2017-05-10 to Sprint 2017-05-24 on the Wikispeech board.May 24 2017, 8:00 AM
Sebastian_Berlin-WMSE edited projects, added Wikispeech (Sprint 2017-05-24); removed Wikispeech (Sprint 2017-05-10).
Sebastian_Berlin-WMSE triaged this task as Medium priority.May 24 2017, 8:04 AM
Sebastian_Berlin-WMSE moved this task from Sprint 2017-05-24 to Sprint 2017-06-07 on the Wikispeech board.Jun 5 2017, 4:24 PM
Sebastian_Berlin-WMSE edited projects, added Wikispeech (Sprint 2017-06-07); removed Wikispeech (Sprint 2017-05-24).
gerritbot added a subscriber: gerritbot.Jun 12 2017, 3:15 PM

Change 358378 had a related patch set uploaded (by Sebastian Berlin (WMSE); owner: Sebastian Berlin (WMSE)):
[mediawiki/extensions/Wikispeech@master] Add API for segmenting text

https://gerrit.wikimedia.org/r/358378

gerritbot added a project: Patch-For-Review.Jun 12 2017, 3:15 PM
Sebastian_Berlin-WMSE mentioned this in rEWIS6d7ee5c8ee65: Add API for segmenting text.Jun 12 2017, 3:15 PM
Sebastian_Berlin-WMSE removed a project: Patch-For-Review.Jun 12 2017, 3:19 PM

Mixed up the task ids.

Sebastian_Berlin-WMSE mentioned this in T146454: WikispeechNamespaces can't contain namspaces that aren't defined as constants.Jun 14 2017, 10:08 AM
Sebastian_Berlin-WMSE moved this task from Backlog to In progress on the Wikispeech (Sprint 2017-06-07) board.Jun 20 2017, 12:57 PM
Sebastian_Berlin-WMSE changed the point value for this task from 20 to 12.Jun 20 2017, 1:02 PM
Sebastian_Berlin-WMSE edited projects, added Wikispeech (Sprint 2017-06-21); removed Wikispeech (Sprint 2017-06-07).

Worked on in Wikispeech (Sprint 2017-06-07):

  • Most of requesting segments, playing/stopping and navigation reimplemented.
  • Started on rewriting tests.

To do in Wikispeech (Sprint 2017-06-21):

  • Rewrite remaining tests.
  • Rebase, comment and clean up code.
Sebastian_Berlin-WMSE moved this task from Backlog to In progress on the Wikispeech (Sprint 2017-06-21) board.Jun 20 2017, 1:03 PM
Lokal_Profil changed the point value for this task from 12 to 8.Jul 4 2017, 9:17 AM
Lokal_Profil edited projects, added Wikispeech (Sprint 2017-07-05); removed Wikispeech (Sprint 2017-06-21).

Worked on in Wikispeech (Sprint 2017-06-21):

  • Tests largely rewritten
  • Re-implemented some logic due to differences in structure to HTML.

To do in Wikispeech (Sprint 2017-07-05):

  • Rebase, comment and clean up code.
Lokal_Profil moved this task from Backlog to In progress on the Wikispeech (Sprint 2017-07-05) board.Jul 4 2017, 9:17 AM
Sebastian_Berlin-WMSE added a comment.Jul 10 2017, 7:32 AM

This is now implemented locally and seem to run fine with the new API. "Just" need to clean up the code now.

Lokal_Profil added a comment.EditedJul 10 2017, 9:00 AM

Not to self from review for previous patch.

  • Check what the default removeTags and segmentBreakingTags are when accessing the API directly. Specifically do they default to the config settings?
  • Check that section [edit] links are handled correctly
Sebastian_Berlin-WMSE closed subtask T164250: Create API action for getting segmented text as Resolved.Jul 11 2017, 3:47 PM
gerritbot added a comment.Jul 17 2017, 9:20 AM

Change 365563 had a related patch set uploaded (by Sebastian Berlin (WMSE); owner: Sebastian Berlin (WMSE)):
[mediawiki/extensions/Wikispeech@master] Use API to retrieve utterances

https://gerrit.wikimedia.org/r/365563

gerritbot added a project: Patch-For-Review.Jul 17 2017, 9:20 AM
Sebastian_Berlin-WMSE mentioned this in rEWISc67c26b95feb: Use API to retrieve utterances.Jul 17 2017, 9:25 AM
Sebastian_Berlin-WMSE moved this task from This Week to Waitin' on the User-Sebastian_Berlin-WMSE board.Jul 17 2017, 9:53 AM
Lokal_Profil mentioned this in rEWISfcd3d105b3b4: Use API to retrieve utterances.Jul 19 2017, 8:42 AM
Lokal_Profil mentioned this in rEWISbe65eb260df6: Use API to retrieve utterances.Aug 4 2017, 2:20 PM
Sebastian_Berlin-WMSE mentioned this in rEWIS407c1f796b44: Use API to retrieve utterances.Aug 24 2017, 2:48 PM
Sebastian_Berlin-WMSE mentioned this in rEWIS1c4575e2910b: Use API to retrieve utterances.Aug 25 2017, 7:51 AM
Sebastian_Berlin-WMSE mentioned this in rEWIS7cfd1ca70f97: Use API to retrieve utterances.Aug 25 2017, 9:19 AM
gerritbot added a comment.Aug 25 2017, 9:32 AM

Change 365563 merged by jenkins-bot:
[mediawiki/extensions/Wikispeech@master] Use API to retrieve utterances

https://gerrit.wikimedia.org/r/365563

Sebastian_Berlin-WMSE closed this task as Resolved.Aug 25 2017, 9:37 AM
Sebastian_Berlin-WMSE moved this task from Waitin' to Done on the User-Sebastian_Berlin-WMSE board.
Sebastian_Berlin-WMSE moved this task from In progress to Done on the Wikispeech (Sprint 2017-07-05) board.
Stashbot mentioned this in T172094: npm 3 requires peer dependencies to be explicitly defined in package.json.Aug 25 2017, 9:59 AM
Stashbot mentioned this in T173516: Use json file extension for the stylelint config file.
Stashbot added a subscriber: Stashbot.

Mentioned in SAL (#wikimedia-cloud) [2017-08-25T09:59:28Z] <Sebastian-WMSE> Deploy latest from Git master: cdedb07, 827fbb2 (T172094), 827fbb2, 2791dad, a7e5eaf, 03ca986, d84db16 (T173516), 7cfd1ca (T164251, T146454)

Diffusion mentioned this in rEWIS49e35e2c6e1b: Update patch set 7.Jun 11 2018, 6:24 AM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL