\***drumroll please**\*

**Introducing Tinypus!**

![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/6324eabf05bd8a54c6eb1650/gJDAcOioOe0nzJLVzPfav.jpeg)

I passthrough merged base Tiny Llama Chat with itself, then fine-tuned with around 1/3 of Platypus dataset.

Observations:

- It's smarter (I think?)

- It sometimes throws "### Instruction:" line. This could be due to the platypus dataset, or the fact that I know jackshit about programming. You can add it to "custom stopping strings" in oobaboga.

- It may be possible to train very specialized mini experts and merge them???

  **Template**
  
  Same with TinyLlama/TinyLlama-1.1B-Chat-v1.0

**Merge details**

slices:
  - sources:
    - model: E://text-generation-webui//models//TinyLlama
      
      layer_range: [0, 12]
      
  - sources:
    
    - model: E://text-generation-webui//models//TinyLlama
      
      layer_range: [4, 22]
    
merge_method: passthrough

dtype: bfloat16

**QLoRA Details**

Chunk Length: 1152
R/A: 64/128
Epoch: 1
q-k-v-o
{"license": "mit", "datasets": ["garage-bAInd/Open-Platypus"], "pipeline_tag": "text-generation"}
transformers safetensors llama text-generation conversational dataset:garage-bAInd/Open-Platypus license:mit autotrain_compatible endpoints_compatible text-generation-inference region:us