Lucie-7B-Instruct / tokenizer.json

Commit History

Fix BOS (should be added by the tokenizer). Fix EOS (in config.json -> must be <|eot_id|>). Remove spaces that were aded after special tags (so that round-trip tests can pass and some bugs avoided when inferring with the model). More complete generation_config.json
e7e496d

Jeronymous commited on

Update /data-server/models/text/llm/multilang/Lucie-7B/instruct/full_instruct/assistant_only/mix_4/transformers_checkpoints/global_step317/tokenizer.json
e86604d
verified

juliehunter commited on