anthonymeo

/

mergekit-ties-yqqvtle

like 0

Why do only this config works in 4bit models but fails with slerp method

models:

• model: anthonymeo/NEC-merged-best
  parameters:
  density: 0.5
  weight: 0.5
• model: anthonymeo/llama3.1-merged
  parameters:
  density: 0.5
  weight: 0.5
  merge_method: ties
  base_model: unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit
  parameters:
  normalize: false
  int8_mask: true
  dtype: float16

slerp config
slices:

• sources:
  • model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
    layer_range:
    • 0
    • 28
  • model: taareshg/Llama-3.2-3B-Instruct-En-Hi-merge-200k
    layer_range:
    • 0
    • 28
      merge_method: slerp
      base_model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
      parameters:
      t:
  • filter: self_attn
    value:
    • 0
    • 0.5
    • 0.3
    • 0.7
    • 1
  • filter: mlp
    value:
    • 1
    • 0.5
    • 0.7
    • 0.3
    • 0
  • value: 0.5
    dtype: bfloat16