How to return more tokens when calling the inference end point?
Hi,I purchased the pro account and spun up a WizardLM/WizardLM-13B-V1.1 end point for text generation.  The API only seems to return a max of 75 characters.  I’ve tried adding max_tokens and min_tokens and return_full_text, it doesn’t seem to have an effect.How can I increase the amount of tokens returned?Here’s an example of what I’ve tried:curl https://abcd.us-east-1.aws.endpoints.huggingface.cloud \
-X POST \
-d '{"inputs":"I want to build a large parking structure.  Can you tell me the steps I would take?", "options": {"min_tokens": 5000, "max_tokens": 5000, "return_full_text": true}}' \
-H "Authorization: Bearer $HF_TOKEN" \
-H "Content-Type: application/json"
2023-07-27T06:08:20Z
Hell@maplesakeplease check thedocumentationand theblog post.
2024-03-29T10:15:50Z
@maplesakeHello. I have the same problem. Have you found a solution?I don’t understand why this is not clearly written in the documentation, there is nothing at all on this subject.
2024-04-21T12:48:45Z
I agree documentation can be much better. max_new_tokens parameter is what you are looking for.
2024-05-09T19:31:11Z
For people finding this thread at a later time. This is an example of how a request (for Llama 3 8B) might look like:{
    "inputs": "<|begin_of_text|><|start_header_id|>user<|end_header_id|>How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>",
    "parameters": {
        "max_new_tokens": 1000
    }
}