Page Menu
Home
Phabricator
Search
Configure Global Search
Log In
Paste
P8479
(An Untitled Masterwork)
Active
Public
Actions
Authored by
EBernhardson
on May 6 2019, 6:37 PM.
Edit Paste
Archive Paste
View Raw File
Subscribe
Mute Notifications
Award Token
Flag For Later
Tags
None
Referenced Files
F28943709: raw.txt
May 6 2019, 6:37 PM
2019-05-06 18:37:01 (UTC+0)
Subscribers
None
public static String[] tokenizeString(String query, String lang) {
GlentTokenizer tokenizer;
if ("ko".equalTo(lang)) {
tokenizer = Tokenizers.korean();
} else if ("ja".equalTo(lang)) {
tokenizer = Tokenizers.japanese();
} else if ("zh".equalTo(lang)) {
tokenizer = Tokenizers.simplifiedChinese();
} else {
return new String[0];
}
return tokenizer.tokenize(query).toArray(new String[0]);
}
Event Timeline
EBernhardson
created this paste.
May 6 2019, 6:37 PM
2019-05-06 18:37:01 (UTC+0)
Log In to Comment