# merged

This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).

## Merge Details
### Merge Method

This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_1_V1](https://huggingface.co/NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_1_V1) as a base.

### Models Merged

The following models were included in the merge:
* [NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_3_V1](https://huggingface.co/NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_3_V1)
* [NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_2_V1](https://huggingface.co/NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_2_V1)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
base_model:
  model:
    path: NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_1_V1
dtype: bfloat16
merge_method: dare_ties
slices:
- sources:
  - layer_range: [0, 22]
    model:
      model:
        path: NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_1_V1
  - layer_range: [0, 22]
    model:
      model:
        path: NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_1_V1
    parameters:
      density: 0.55
      weight: 0.55
  - layer_range: [0, 22]
    model:
      model:
        path: NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_2_V1
    parameters:
      density: 0.55
      weight: 0.56
  - layer_range: [0, 22]
    model:
      model:
        path: NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_3_V1
    parameters:
      density: 0.55
      weight: 0.56
```



```Python
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging,
    GenerationConfig,
    TextIteratorStreamer,
)
import torch

new_model= "NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_all_Cluster_merge_v1"
model = AutoModelForCausalLM.from_pretrained(#f'NickyNicky/{new_model}',
                                             new_model,
                                             device_map="auto",
                                             trust_remote_code=True,
                                             torch_dtype=torch.bfloat16,

                                             low_cpu_mem_usage= True,
                                            #  use_flash_attention_2=False,

                                             )


tokenizer = AutoTokenizer.from_pretrained(new_model,
                                          max_length=2048,
                                          trust_remote_code=True,
                                          use_fast = True,
                                          )

tokenizer.pad_token = tokenizer.eos_token
# tokenizer.padding_side = 'left'
tokenizer.padding_side = 'right'


prompt= """<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
escribe una historia de amor.<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer.encode(prompt,
                          return_tensors="pt",
                          add_special_tokens=False).cuda()#.to("cuda") # False # True


generation_config = GenerationConfig(
              max_new_tokens=700,
              temperature=0.5,
              top_p=0.9,
              top_k=40,
              repetition_penalty=1.1, #1.1, # 1.0 means no penalty, > 1.0 means penalty, 1.2 from CTRL paper
              do_sample=True,
              pad_token_id=tokenizer.eos_token_id,
              eos_token_id=tokenizer.eos_token_id,
          )
outputs = model.generate(
                         generation_config=generation_config,
                         input_ids=inputs,)
# tokenizer.decode(outputs[0], skip_special_tokens=False) #True
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
```
{"language": ["en", "es", "ru", "zh", "de", "fr", "th", "ca", "it", "ja", "pl", "eo", "eu", "vi", "fi", "hu", "ar", "nl", "da", "tr", "ko", "he", "id", "cs", "bn", "sv"], "license": "apache-2.0", "tags": ["mergekit", "merge"], "base_model": ["NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_1_V1", "NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_3_V1", "NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_2_V1"], "widget": [{"text": "<|im_start|>system\nYou are a helpful AI assistant.<|im_end|>\n<|im_start|>user\npodrias escribir un codigo de ejemplo en Python<|im_end|>\n<|im_start|>assistant\n"}]}
transformers safetensors llama text-generation mergekit merge en es ru zh de fr th ca it ja pl eo eu vi fi hu ar nl da tr ko he id cs bn sv arxiv:2311.03099 arxiv:2306.01708 base_model:NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_1_V1 base_model:NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_3_V1 base_model:NickyNicky/TinyDolphin-2.8-1.1b_oasst2_chatML_Cluster_2_V1 license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us