2026-10-14T16:01:29Z https://www.scidb.cn/oai

10.11922/sciencedb.o00114.00060 2021-12-10T15:10:00Z

2021-12-10 A Method of Tokenization Based on Neural Network Natural Language Processing Model doi:10.11922/sciencedb.o00114.00060 en This report will review the Tokenization method in natural language processing based on neural networks. First, I will explain the out-of-vocabulary (OOV) problem caused by the closed vocabulary in the natural language processing based on neural network, and introduce two common methods for solving this problem, BPE and WordPiece, and a derivative method, SentencePiece technology. Although BPE is simple and efficient, there might be a insufficient learning problem of some sub-words. In that way, I will introduce BPE Dropout technology to solve this problem. Character-based BBPE and WordPiece still have out-of-vocabulary(OOV) problem when facing mult-languages large character sets (especially CJK languages). I will introduce an effective method to solve this problem: a tokenization technology based on UTF-8 characters (mainly BBPE), and its derivative technology BBPE-based SentencePiece. Finally, I will introduce VOLT, a general vocabulary size optimization technique proposed by ACL2021 best paper. Tokenization; Neural network; Review Qun Liu PUBLIC https://creativecommons.org/licenses/by/4.0/ dataset Science Data Bank