Available sizes

Branch	Bits	Description
8_0	8.0	Maximum quality that ExLlamaV2 can produce, near unquantized performance.
6_5	6.5	Very similar to 8.0, good tradeoff of size vs performance, recommended.
5_0	5.0	Slightly lower quality vs 6.5, but usable
4_25	4.25	GPTQ equivalent bits per weight, slightly higher quality.
3_5	3.5	Lower quality, only use if you have to.

Download instructions

With git:

git clone --single-branch --branch 6_5 https://huggingface.co/bunnycore_-_Qwen-2.5-3b-Evol-CoT-exl2 Qwen-2.5-3b-Evol-CoT-6_5

With huggingface hub:

pip3 install huggingface-hub

To download a specific branch, use the --revision parameter. For example, to download the 6.5 bpw branch: Linux:

huggingface-cli download bunnycore_-_Qwen-2.5-3b-Evol-CoT-exl2 --revision 6_5 --local-dir Qwen-2.5-3b-Evol-CoT-6_5 --local-dir-use-symlinks False

Windows (which apparently doesn't like _ in folders sometimes?):

huggingface-cli download bunnycore_-_Qwen-2.5-3b-Evol-CoT-exl2 --revision 6_5 --local-dir Qwen-2.5-3b-Evol-CoT-6.5 --local-dir-use-symlinks False

Original model description:

base_model:

bunnycore/Qwen-2.5-3b-RP
Replete-AI/Replete-LLM-V2.5-Qwen-3b
bunnycore/Qwen-2.5-3b-Mix-Data-lora
bunnycore/Qwen-2.5-3b-RP
bunnycore/Qwen-2.5-3b-Mix-Data-lora library_name: transformers tags:
mergekit
merge

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using bunnycore/Qwen-2.5-3b-RP as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Replete-AI/Replete-LLM-V2.5-Qwen-3b+bunnycore/Qwen-2.5-3b-Mix-Data-lora
    parameters:
      density: 0.5
      weight: 0.5
  - model: bunnycore/Qwen-2.5-3b-RP+bunnycore/Qwen-2.5-3b-Mix-Data-lora
    parameters:
      density: 0.5
      weight: 0.5

merge_method: dare_ties
base_model: bunnycore/Qwen-2.5-3b-RP
parameters:
  normalize: false
  int8_mask: true
dtype: float16