A Method of Tokenization Based on Neural Network Natural Language Processing Model

First, I will explain the out-of-vocabulary (OOV) problem caused by the closed vocabulary in the natural language processing based on neural network, and introduce two common methods for solving this problem, BPE and WordPiece, and a derivative method, SentencePiece technology. Although BPE is simple and efficient, there might be a insufficient learning problem of some sub-words.

In that way, I will introduce BPE Dropout technology to solve this problem. Character-based BBPE and WordPiece still have out-of-vocabulary(OOV) problem when facing mult-languages large character sets (especially CJK languages). I will introduce an effective method to solve this problem: a tokenization technology based on UTF-8 characters (mainly BBPE), and its derivative technology BBPE-based SentencePiece.