Refactor Trainer and pretokenization interface#1287

taku910 merged 2 commits into

mastergoogle/sentencepiece:masterfrom

trainergoogle/sentencepiece:trainerCopy head branch name to clipboard