How do you use SentencePiece for BPE of sequences with no whitespace
I am trying to use byte pair encoding on amino acid sequences which have no spaces:ADNRRPIWNLGHMVNALKQIPTFLXDGANAthe tokenizers summary section of the docs states suggests SentencePiece could be useful, as it treats the input as a raw stream, includes the space in the set of characters to use, then uses BPE or unigram to construct the appropriate vocabulary.How would I train a tokenizer from scratch using SentencePiece? The tokenizer library seems to only supportWordPiece.
2021-04-29T16:50:59Z
In original sentencepiece model, white space is considered as a regular character. Please read the description here.GitHubGitHub - google/sentencepiece: Unsupervised text tokenizer for Neural...Unsupervised text tokenizer for Neural Network-based text generation. - GitHub - google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation.I am not totally familier with the huggingface implementation of the sentencepiece. But you can use the original sentencepiece library for that and then try loading that sentencepiece model by huggingface wrapper if needed.