Yunus Manti's picture

Yunus Manti

Yuma42

·

AI & ML interests

I'm generating and running AI to experiment and educate myself.

Recent Activity

replied to bartowski's post 13 days ago

Looks like Q4_0_N_M file types are going away Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable) You can see the reference PR here: https://github.com/ggerganov/llama.cpp/pull/10446 So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms) As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those ! Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541 Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights

reacted to bartowski's post with 👍 13 days ago

Looks like Q4_0_N_M file types are going away Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable) You can see the reference PR here: https://github.com/ggerganov/llama.cpp/pull/10446 So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms) As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those ! Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541 Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights

new activity 29 days ago

open-llm-leaderboard/open_llm_leaderboard:Proposal for new column

View all activity

Organizations

None yet

Yuma42's activity

New activity in open-llm-leaderboard/open_llm_leaderboard 29 days ago

Proposal for new column

#1032 opened 29 days ago by

New activity in Joseph717171/Llama-3.1-SuperNova-8B-Lite_TIES_with_Base 2 months ago

Q4_0_4_4 gguf

#3 opened 2 months ago by

New activity in rombodawg/Rombos-LLM-V2.5-Qwen-7b 3 months ago

Chat template

#1 opened 3 months ago by

New activity in bartowski/Hermes-3-Llama-3.1-8B-lorablated-GGUF 4 months ago

Q4_0_4_4

#2 opened 4 months ago by

New activity in open-llm-leaderboard/open_llm_leaderboard 4 months ago

Question: same model with very different scores

#904 opened 4 months ago by

New activity in NousResearch/Hermes-2-Theta-Llama-3-8B-GGUF 5 months ago

More quantization variants

#1 opened 8 months ago by

New activity in open-llm-leaderboard/open_llm_leaderboard 5 months ago

Gemma-2-9B-it scores

#843 opened 6 months ago by

New activity in Yuma42/KangalKhan-RawRuby-7B 5 months ago

Adding Evaluation Results

#2 opened 5 months ago by

leaderboard-pr-bot

New activity in open-llm-leaderboard/open_llm_leaderboard 6 months ago

Bug with gui

#802 opened 6 months ago by

Upvote to evaluate deepseek-coder-v2

#793 opened 7 months ago by

New activity in mlabonne/NeuralDaredevil-8B-abliterated 7 months ago

Couldn't find on the leaderboard.

#2 opened 7 months ago by deleted

New activity in bartowski/Hermes-2-Theta-Llama-3-8B-GGUF 8 months ago

Prompt format correct?

#1 opened 8 months ago by

New activity in TIGER-Lab/MMLU-Pro 8 months ago

Benchmark request

#4 opened 8 months ago by

New activity in mradermacher/KangalKhan-RawRuby-7B-GGUF 8 months ago

Thank you again + Request

#1 opened 8 months ago by

New activity in LDJnr/Capybara 8 months ago

Thank you

#8 opened 8 months ago by

New activity in argilla/dpo-mix-7k 8 months ago

Thank you

#5 opened 8 months ago by

New activity in mradermacher/KangalKhan-NeoRuby-7B-GGUF 8 months ago

Wow thanks

#1 opened 8 months ago by

New activity in open-llm-leaderboard/open_llm_leaderboard 9 months ago

Models not evaluated

#655 opened 9 months ago by

New activity in Yuma42/KangalKhan-ShatteredRuby-7B 10 months ago

Adding Evaluation Results

#1 opened 10 months ago by

leaderboard-pr-bot

New activity in Yuma42/KangalKhan-PrimordialSapphire-7B 10 months ago

Adding Evaluation Results

#1 opened 10 months ago by

leaderboard-pr-bot