Maniphest T328267

Word Tokenization: Non-whitespace languages
Closed, DeclinedPublic
Actions

Assigned To

Authored By

	Appledora
	Jan 30 2023, 9:41 AM

Project Tags

Referenced Files

None

Subscribers

Description

Identify languages not following white-space based word tokenization schemes
Collect corpus for unsupervised training
Setup sentencepiece training environment

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Appledora	T316941 NLP Tools for Content Gaps
		Resolved		Appledora	T328264 NLP Tools: Word Tokenization
		Declined		Appledora	T328267 Word Tokenization: Non-whitespace languages
		Declined		Appledora	T328269 Sentencepiece: Language Family Wise training
		Declined		Appledora	T328270 Sentencepiece: all non-whitespace languages

Event Timeline

Appledora created this task.Jan 30 2023, 9:41 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 30 2023, 9:41 AM

now tracked under T328264

Isaac closed subtask T328269: Sentencepiece: Language Family Wise training as Declined.Feb 20 2023, 5:23 PM

Isaac closed subtask T328270: Sentencepiece: all non-whitespace languages as Declined.