text
So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
new_line
new_line
text
It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
new_line
new_line
text
However what the quantization then does with that information is where I was wrong.
However what the quantization then does with that information is where I was wrong.
new_line
new_line
text
I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
new_line
new_line
text
Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
new_line
new_line
text
The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
new_line
new_line
text
But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
new_line
new_line
text
Huge shoutout to 
Huge shoutout to 
mention
@compilade
compilade
text
 for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
 for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp

It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important

However what the quantization then does with that information is where I was wrong.

I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW

Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations

The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group

But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix

Huge shoutout to @compilade for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
❤️
MaziyarPanahi victor Themoneybuddha John6666 bullerwins Rybens Anderson452 NOBODYVONNOTHING qnixsynapse Clausss exdysa AtAndDev IAmTheCollector wsuff rreed-pha djuna MarinaraSpaghetti Pomni RakshitAralimatti osanseviero venketh Arakinas wonderboy nicoboss Joseph717171 AIGUYCONTENT Handgun1773 bradeenl EloyOn
👍
Aurelien-Morgan Clausss AtAndDev sroecker iandol rreed-pha ajibawa-2023 ijohn07 thiera1 nbeerbower newsletter venketh mpk63 wonderboy clem nicoboss jsfs11 Joseph717171 waldie fendiprime
🚀
rreed-pha wonderboy nicoboss Joseph717171
https://cdn-avatars.huggingface.co/v1/production/uploads/5fd5e18a90b6dc4633f6d292/gZXHW5dd9R86AV9LMZ--y.png
Maziyar Panahi
MaziyarPanahi
user
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_O-ycBVEVTAvhqC9wH4v2.png
Eric Sims
Themoneybuddha
user
/avatars/c6c710651ec2bde1680497afd4649281.svg
John Freier
mpk63
user
https://cdn-avatars.huggingface.co/v1/production/uploads/623aada982038110d90921f3/nSHSXJz7PYyw9fn8qoAn4.jpeg
zero
wonderboy
user
/avatars/ea4398745974d781ae9dc0e95b12cabe.svg
Joseph
Joseph717171
user