Problem for large context window (400k)
Hi,I’d like to use the model namespace-Pt/activation-beacon-llama2-7b-chat. It’s a model with a 400k context window, which is what I want.But when I try to launch a dedicated Inference Endpoints, when I use it, it says {‘error’: 'Input validation error:inputstokens +max_new_tokensmust be <= 2048. […] }.I already tried to modify Max Input Length (per Query) and Max Number of Tokens (per Query), but I have the same problem.So how can I use this large context window model please ?Thank you !
2024-05-02T11:38:21Z
I need help!!!
2024-06-27T14:59:19Z
No help, but just to let you know: I have a similar problem with the gradient llama3 models, I need a big context window, but the inference endpoint somehow does not play along nicely. Asked support, no answer yet.
2024-07-02T17:28:41Z
Hi also want to know!
2024-07-24T08:20:20Z
Same here. I’m chasing my tail trying to deploy a large context window model on an inference endpoint.Some general guidance on how to do it would be nice.