Issue with german umlauts python in deepseek-ai/deepseek-coder-1.3b-instruct
Hi,I have an issue with German umlauts python in deepseek-ai/deepseek-coder-1.3b-instruct tokenizer.If I run the following script I got � für every German umlaut.from transformers import AutoTokenizer
import torch
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
print("Target Device: ", DEVICE)
MODEL = "deepseek-ai/deepseek-coder-1.3b-instruct"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
print (tokenizer)
inputs = tokenizer("A", return_tensors="pt").to("cuda")
inputs.input_ids[0][1] = 32009
print (inputs)
decoded = tokenizer.decode(inputs.input_ids[0])
print (decoded)2024-02-16 13_01_46-michael@orin_ ~_fastchat1553×526 74.4 KBHow to fix this?