Decode token IDs into a list (not a single string)
tokenizer.convert_ids_to_tokensreturns:['ĠDrive', 'Ġwas', 'Ġhad', 'Ġwalked', "'s", ',', 'Ġlooked', ...]I need the tokens without the special characters.decodedoesnotwork, because it only returns a single string.Is there a function that outputs the plain tokens as a list?
2023-06-22T07:11:37Z
Hey! Not sure I completely understand, but the tokens that you have here are theplaintokens, as they are in the vocab / merge. You should modify the tokenizer if you do not want it to add thespiecetoken at the beginning. Which tokenizer are you using?
2023-06-23T03:40:18Z
Thanks for the ping!I was using the GPT byte level tokenizer.I’m not sure if this is a hack, but to get the behavior I wanted, I just passed the token ids intodecode_batchinstead, and that returned each token without the odd encoding.
2023-09-18T21:17:43Z
It’s not a hack, but something I wish to improve! IMObatch_decodeanddecodeshould be merged into one as we only haveencode