Lucie-7B-Instruct / tokenizer_config.json

Commit History

Fix BOS (should be added by the tokenizer). Fix EOS (in config.json -> must be <|eot_id|>). Remove spaces that were aded after special tags (so that round-trip tests can pass and some bugs avoided when inferring with the model). More complete generation_config.json
e7e496d

Jeronymous commited on

Update /data-server/models/text/llm/multilang/Lucie-7B/instruct/full_instruct/assistant_only/mix_4/transformers_checkpoints/global_step317/tokenizer_config.json
f8df234
verified

juliehunter commited on