Page MenuHomePhabricator
Create Task
Maniphest T261928

☂Wikispeech Recording Manuscript Tool
Closed, ResolvedPublic
Actions

Description

Set of tools for extracting text (sentences) suitable for recording from a large text corpus (typically a Wikipedia dump file or similar).

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNikolajLindbergT261928 ☂Wikispeech Recording Manuscript Tool
 ResolvedNikolajLindbergT261929 Indexed Relational Database of Text
 ResolvedNikolajLindbergT261930 Add article and its sentences in one single transaction
ResolvedNikolajLindbergT261934 Filtering function of text
 ResolvedNikolajLindbergT262011 Filter out features ("sentences without digits", etc)
 ResolvedNikolajLindbergT261955 Add source feature: article length in paragraphs
ResolvedNikolajLindbergT262010 dbapi.Add(text.Article)
 OpenNoneT262014 Add tests for lookup
 OpenNoneT262015 dbapi: clean up and remove/un-export some exposed functions
 ResolvedNikolajLindbergT262115 load_db tool should filter articles and sentences before adding to db
OpenNoneT262116 new db chunkfeat type: Unicode code block
 ResolvedNikolajLindbergT262117 generated word frequency table
 ResolvedNikolajLindbergT262121 chunkfeat punct as single characters not character sequence
 ResolvedHannaLindgrenT261935 Selection algorithm for creating a balanced manuscript from a set of sentences
 DeclinedHannaLindgrenT265599 Final evaluation of selection algorithm
 ResolvedHannaLindgrenT265602 New selection opt type using target amount
ResolvedNikolajLindbergT261938 Find out how to best extract raw text from Wikipedia articles (probably from a dump file)
 ResolvedNikolajLindbergT262131 Figure out why two different runs of WikiExtractor.py used different paragraph delimiters?
 OpenNoneT262199 Improve regexp for splitting paragraphs into sentences
DeclinedNikolajLindbergT261931 Create simplistic web GUI for basic filtering
 InvalidNoneT265598 Optimize generate_db_featcats
 ResolvedNikolajLindbergT265600 Add extracted featcats to Sentence struct
ResolvedHannaLindgrenT265601 Define feature names as constants
 ResolvedHannaLindgrenT265604 Blocking sentences
 DeclinedHaraldBerthelsenT265608 Check warnings from WikiExtractor.py
 ResolvedHannaLindgrenT265630 Score computation should take previously selected features into account
 ResolvedNikolajLindbergT265631 Create UML diagram for manuscript database
 ResolvedHannaLindgrenT265730 New features: final trigrams, initial bigrams
 ResolvedHannaLindgrenT265731 New selection algorithm should compute score by random chunks of variable size
 InvalidBUG REPORTNikolajLindbergT265736 WriteScriptToDB does not save to db
 InvalidBUG REPORTNikolajLindbergT265758 Basic filter finds some duplicate ids
 ResolvedHannaLindgrenT266018 Combined API (CLI) to use filter and selection algorithm in one application
 ResolvedHannaLindgrenT266764 Refactoring filter and selection
 ResolvedHannaLindgrenT266792 New db tables: batch_properties and script_properties
 ResolvedNikolajLindbergT267119 Facilitate different runs with different scoring functions (in sequence or as separate batches)
 ResolvedNikolajLindbergT267802 Documentation of manuscript process and algorithm
 ResolvedHannaLindgrenT269100 Generate various manuscripts for demonstration
 ResolvedHannaLindgrenT272360 Publish manuscript source code on github

Event Timeline

NikolajLindberg created this task.Sep 3 2020, 9:21 AM
NikolajLindberg moved this task from Unsorted to Manuscript creator on the Wikispeech-Speech-Data-Collector board.Sep 3 2020, 9:33 AM
NikolajLindberg moved this task from Incoming to Sprint on the Wikispeech-Jobrunner board.
NikolajLindberg edited projects, added Wikispeech-Jobrunner (Sprint); removed Wikispeech-Jobrunner.
NikolajLindberg moved this task from Backlog to In progress on the Wikispeech-Jobrunner (Sprint) board.
Lokal_Profil renamed this task from Wikispeech Recording Manuscript Tool to ☂Wikispeech Recording Manuscript Tool.Sep 3 2020, 11:14 AM
NikolajLindberg added a subtask: T261931: Create simplistic web GUI for basic filtering.Sep 17 2020, 10:58 AM
NikolajLindberg closed subtask T261938: Find out how to best extract raw text from Wikipedia articles (probably from a dump file) as Resolved.Oct 1 2020, 1:03 PM
NikolajLindberg closed subtask T261929: Indexed Relational Database of Text as Resolved.
HannaLindgren added a subtask: T265598: Optimize generate_db_featcats.Oct 15 2020, 1:32 PM
HannaLindgren added a subtask: T265599: Final evaluation of selection algorithm.
HannaLindgren closed subtask T265601: Define feature names as constants as Resolved.Oct 15 2020, 2:08 PM
HannaLindgren subscribed.Oct 16 2020, 3:14 PM
HannaLindgren closed subtask T265758: Basic filter finds some duplicate ids as Invalid.Oct 16 2020, 8:06 PM
HannaLindgren closed subtask T265736: WriteScriptToDB does not save to db as Invalid.
HannaLindgren closed subtask T265598: Optimize generate_db_featcats as Invalid.Oct 21 2020, 12:34 PM
HannaLindgren closed subtask T265602: New selection opt type using target amount as Resolved.Oct 29 2020, 3:12 PM
HannaLindgren closed subtask T265604: Blocking sentences as Resolved.
HannaLindgren closed subtask T265631: Create UML diagram for manuscript database as Resolved.
HannaLindgren closed subtask T265630: Score computation should take previously selected features into account as Resolved.
HannaLindgren closed subtask T265731: New selection algorithm should compute score by random chunks of variable size as Resolved.
HannaLindgren closed subtask T265600: Add extracted featcats to Sentence struct as Resolved.
HannaLindgren closed subtask T261935: Selection algorithm for creating a balanced manuscript from a set of sentences as Resolved.
HannaLindgren closed subtask T265730: New features: final trigrams, initial bigrams as Resolved.
HannaLindgren closed subtask T267119: Facilitate different runs with different scoring functions (in sequence or as separate batches) as Resolved.Nov 12 2020, 2:45 PM
HannaLindgren closed subtask T261931: Create simplistic web GUI for basic filtering as Declined.Feb 12 2021, 12:05 PM
HannaLindgren closed subtask T265608: Check warnings from WikiExtractor.py as Declined.Apr 6 2021, 8:48 AM
HannaLindgren closed subtask T266018: Combined API (CLI) to use filter and selection algorithm in one application as Resolved.Apr 8 2021, 11:24 AM
HannaLindgren moved this task from In progress to Backlog on the Wikispeech-Jobrunner (Sprint) board.Apr 13 2021, 8:12 AM
HannaLindgren closed subtask T265599: Final evaluation of selection algorithm as Declined.Apr 13 2021, 10:42 AM
HannaLindgren moved this task from Backlog to Done on the Wikispeech-Jobrunner (Sprint) board.Apr 20 2021, 4:11 PM
HannaLindgren closed subtask T272360: Publish manuscript source code on github as Resolved.Apr 22 2021, 11:52 AM
HannaLindgren closed subtask T267802: Documentation of manuscript process and algorithm as Resolved.
HannaLindgren closed subtask T269100: Generate various manuscripts for demonstration as Resolved.
HannaLindgren closed this task as Resolved.Apr 22 2021, 11:54 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL