mradermacher

/

HuatuoGPT-o1-72B-i1-GGUF

like 0

I'm getting the following error when trying to run Q6_K in Oobabooga:

00:13:52-002812 INFO Loading "HuatuoGPT-QWEN-o1-72B.i1-Q6_K.gguf.part1of2"
00:13:52-234092 INFO llama.cpp weights detected:
"models/HuatuoGPT-QWEN-o1-72B.i1-Q6_K.gguf.part1of2"
llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23177 MiB free
llama_load_model_from_file: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23982 MiB free
llama_load_model_from_file: using device CUDA2 (NVIDIA GeForce RTX 3090) - 23982 MiB free
llama_model_load: error loading model: tensor 'blk.38.ffn_gate.weight' data is not within the file bounds, model is corrupted or incomplete
llama_load_model_from_file: failed to load model
00:13:52-272489 ERROR Failed to load the model.
Traceback (most recent call last):
File "/home/me/text-generation-webui/modules/ui_model_menu.py", line 231, in load_model_wrapper
shared.model, shared.tokenizer = load_model(selected_model, loader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/me/text-generation-webui/modules/models.py", line 93, in load_model
output = load_func_maploader
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/me/text-generation-webui/modules/models.py", line 278, in llamacpp_loader
model, tokenizer = LlamaCppModel.from_pretrained(model_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/me/text-generation-webui/modules/llamacpp_model.py", line 111, in from_pretrained
result.model = Llama(**params)
^^^^^^^^^^^^^^^
File "/home/me/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda_tensorcores/llama.py", line 369, in init
internals.LlamaModel(
File "/home/me/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda_tensorcores/_internals.py", line 56, in init
raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: models/HuatuoGPT-QWEN-o1-72B.i1-Q6_K.gguf.part1of2

Exception ignored in: <function LlamaCppModel.__del__ at 0x73c9083c4c20>
Traceback (most recent call last):
File "/home/me/text-generation-webui/modules/llamacpp_model.py", line 62, in del
del self.model
^^^^^^^^^^
AttributeError: 'LlamaCppModel' object has no attribute 'model'

I have 4x3090, EPYC 7532, ROMED8-2T.

Stock settings except for:

Loader: llama.cpp
n_ctx: 4096
tensor_split: 21,23,23,23
n-gpu-layers: 65
no-mmap
flash_attn
tensorcores
cache_type: fp16

I added the "QWEN" word in both file names to remind myself to use ChatML as the instruction template.

EDIT: I'm getting the same error for: HuatuoGPT-o1-72B.Q5_K_M.gguf

Wonder if I'm doing something wrong? I haven't played with Oobabooga much since the redesign.