Paste P8479

(An Untitled Masterwork)
ActivePublic
Actions

Authored by EBernhardson on May 6 2019, 6:37 PM.

Tags

None

Referenced Files

	F28943709: raw.txt
	May 6 2019, 6:37 PM

Subscribers

None

	public static String[] tokenizeString(String query, String lang) {
	GlentTokenizer tokenizer;
	if ("ko".equalTo(lang)) {
	tokenizer = Tokenizers.korean();
	} else if ("ja".equalTo(lang)) {
	tokenizer = Tokenizers.japanese();
	} else if ("zh".equalTo(lang)) {
	tokenizer = Tokenizers.simplifiedChinese();
	} else {
	return new String[0];
	}
	return tokenizer.tokenize(query).toArray(new String[0]);
	}

Event Timeline

EBernhardson created this paste.May 6 2019, 6:37 PM