Currently we have some basic test functions for word/sentence tokenizations. But they only contain English examples. We need to update these to accommodate a few more scripts.
Description
Description
Currently we have some basic test functions for word/sentence tokenizations. But they only contain English examples. We need to update these to accommodate a few more scripts.