Truncated output on mistralai/Mistral-7B-Instruct-v0.1
I am using an inference endpoint onmistralai/Mistral-7B-Instruct-v0.1. The output is truncated, for instance, the proposed test query "Can you please let us know more details about your " yields “2019 Honda CR-V Touring?\n\n1. What is the mile”. How can I adjust the output size?
2023-12-20T19:23:42Z
Have you tried modifying the ‘max_tokens’ parameter
2023-12-20T20:06:55Z
No I haven’t, is this parameter documented somewhere ? Using the following yields the same result.output = query({
  "inputs": "Can you please let us know more details about your ",
  "parameters": {
    "max_tokens": 128
  }
})
2023-12-20T20:56:11Z
You need the correct prompt template for the model<s>[INST] Can you please let us know more details about your  [/INST]I recommend using outpython clientplusprompt templatesvia transformersHere is a code snippet example:pip install transformers jinja2 huggingface-hubfrom transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import InferenceClient

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
client = InferenceClient("mistralai/Mistral-7B-Instruct-v0.1")

messages = [
    {"role": "user", "content": "Can you please let us know more details about your "},
]

prompt_encoded = tokenizer.apply_chat_template(messages, tokenize=False)
output = client.text_generation(prompt_encoded, max_new_tokens=200)
print(output)Model can be an inference endpointmodel (str, optional) — The model to run inference with. Can be a model id hosted on the Hugging Face Hub, e.g. bigcode/starcoder or a URL to a deployed Inference Endpoint. Defaults to None, in which case a recommended model is automatically selected for the task.
2023-12-21T09:29:21Z
Ok I see, it works now with the prompt template. Thanks!