Number of tokens (2331) exceeded maximum context length (512) error.Even when model supports 8k Context length
Ive loaded Mistral7b on aws sagemaker with context length 8k.But im getting error ;Number of tokens (2332) exceeded maximum context length (512).on line :print(llm("""{Something with 3000 tokens}"""))with context length of 8k tokens of the model , how can  number of tokens (2332) exceeded maximum context length (512) error arise.The code im using is ;imports :# Base ctransformers with no GPU acceleration
!pip install ctransformers
# Or with CUDA GPU acceleration
!pip install ctransformers[cuda]
# Or with AMD ROCm GPU acceleration (Linux only)
!CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
# Or with Metal GPU acceleration for macOS systems only
!CT_METAL=1 pip install ctransformers --no-binary ctransformersCode to load and run model ;from ctransformers import AutoModelForCausalLM

# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Mistral-7B-v0.1-GGUF", model_file="mistral-7b-v0.1.Q5_K_M.gguf", model_type="mistral",gpu_layers=0)
print(llm("""{Something with 3000 tokens}"""))Im using AWS Sagemaker Studio lab notebook that provides CPU  :instance - t3.xlargevCPUs - 4memory - 16GB
2023-10-12T13:17:17Z
You can expand the context length with a config parameterYou could try thisfrom ctransformers import AutoModelForCausalLM,AutoConfig

config = AutoConfig.from_pretrained("TheBloke/Mistral-7B-v0.1-GGUF")
# Explicitly set the max_seq_len
config.max_seq_len = 4096
config.max_answer_len= 1024

# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Mistral-7B-v0.1-GGUF", model_file="mistral-7b-v0.1.Q5_K_M.gguf", model_type="mistral",gpu_layers=0, config=config)
print(llm("""{Something with 3000 tokens}"""))
2023-10-21T01:30:50Z
I have the same issue even using above scripts. Does anybody find a solution? Here is my testing code and results.from ctransformers import AutoModelForCausalLM,AutoConfigmodel_name= “TheBloke/Mistral-7B-Instruct-v0.1-GGUF”model_path = r’D:\Mistral\mistral-7b-instruct-v0.1.Q6_K.gguf’config = AutoConfig.from_pretrained(model_name)config.max_seq_len = 4096config.max_answer_len= 1024llm = AutoModelForCausalLM.from_pretrained(model_name, model_file=model_path, model_type=“mistral”,gpu_layers=0, config=config)prompt = “”"Please summarize below article in one sentence.####… 600 words“”"print(llm(prompt))(myenv) C:\myenv\test >python C:\myenv\test\mytest.py…Number of tokens (893) exceeded maximum context length (512).Number of tokens (894) exceeded maximum context length (512).Number of tokens (895) exceeded maximum context length (512).{‘input’: 'Summary: ‘, ‘text’: ’ 10006 7, 8, 4 3\n\t2121212983’}duration: 119.195716
2023-11-02T16:59:51Z
Set config like belowconfig.config.max_new_tokens = 2048config.config.context_length = 4096It solved the issue for me.
2023-11-03T03:55:09Z
It does not work for me either. I got “Number of tokens (692) exceeded maximum context length (512)”.
2023-11-15T20:48:21Z
This worked for me:llm = AutoModelForCausalLM.from_pretrained("TheBloke/zephyr-7B-beta-GGUF", 
                                           model_file="zephyr-7b-beta.Q5_K_M.gguf", 
                                           model_type="mistral", 
                                           gpu_layers=50,
                                           max_new_tokens = 1000,
                                           context_length = 6000)No warnings output.
2023-12-25T08:38:26Z
llm = AutoModelForCausalLM.from_pretrained(“TheBloke/Llama-2-7B-GGUF”, model_file=“llama-2-7b.Q6_K.gguf”, model_type=“llama”,context_length=4096, max_new_tokens=4096, gpu_layers=0)this worked for me
2023-12-25T08:39:34Z
Also, I think I read somewhere that ctransformers only support Llama and 2 other base model for context changes. Please correct me if I am wrong.
2024-10-06T17:07:20Z
I do the following for a model with whose context_size = 2048 and I still get the error that says‘’’ A single document was longer than the context length, we cannot handle this.‘’’But, the maximum token length among the documents is only 1024 and along with each document all I am sending is the question ‘What are conditions for termination’from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig

config = AutoConfig.from_pretrained('amd/AMD-Llama-135m')
config.context_length = 2048

tokenizer = AutoTokenizer.from_pretrained("amd/AMD-Llama-135m", truncation = True)
model = AutoModelForCausalLM.from_pretrained("amd/AMD-Llama-135m")Note : config.config.context_length throws AtrributeError@Rockey1@Professor-HuntScreenshot (21)_LI1036×988 79.3 KB