

# Model description
aguila-7b-Qlora-instruct is a transformer-based addapter causal language model for Catalan, Spanish, and English trained with [Guanaco Spanish dataset](https://huggingface.co/datasets/hlhdatscience/guanaco-spanish-dataset) to be an instruction model.
This model contains the Qlora adapters necessary to use the model.
It is based on the [Ǎguila-7B model](https://huggingface.co/projecte-aina/aguila-7b), that  has been trained on a 26B token trilingual corpus collected from publicly available corpora and crawlers.

More information available in the following post from Medium.com: Introducing Ǎguila, a new open-source LLM for Spanish and Catalan (https://medium.com/@mpamies247/introducing-a%CC%8Cguila-a-new-open-source-llm-for-spanish-and-catalan-ee1ebc70bc79)

Intended uses and limitations
The Ǎguila-7B-Qlora-Instruct model is ready-to-use with qlora adapters for assistant porpouses. 

# How to use
Here is how to use this model:

```python
import torch
from peft import PeftConfig, PeftModel
from transformers import GenerationConfig, AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

MODEL_NAME = "hlhdatscience/aguila-7b-Qlora-instruct"
BASE_MODEL = "projecte-aina/aguila-7b"

compute_dtype = getattr(torch, "float16")

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=compute_dtype,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

model = PeftModel.from_pretrained(model, MODEL_NAME)


generation_config = model.generation_config
generation_config.top_p = 0.7
generation_config.num_return_sequences = 1
generation_config.max_new_tokens = 150
generation_config.use_cache = True
generation_config.pad_token_id = tokenizer.eos_token_id
generation_config.eos_token_id = tokenizer.eos_token_id

model.eval()
'''if torch.__version__ >= "2":
    model = torch.compile(model)'''  # OPTIONAL

def generate_prompt(prompt: str) -> str:
    return f"""
    ### humano: {prompt}
    ### asistente: 
    """.strip()

prompt = "¿Qué es la guerra?"

prompt = generate_prompt(prompt)
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
input_ids = input_ids.to(model.device)

with torch.no_grad():
    outputs = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
    )

response = tokenizer.decode(outputs.sequences[0], skip_special_tokens=True)
print(response)

torch.cuda.empty_cache()
```

# Original information from the Base Model

## Limitations and bias
At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model. 
However, we are well aware that our models may be biased since the corpora have been collected using crawling techniques 
on multiple web sources. We intend to conduct research in these areas in the future, and if completed, this model card will be updated. 


## Language adaptation

We adapted the original [Falcon-7B](https://huggingface.co/tiiuae/falcon-7b) model to Spanish and Catalan by swapping the tokenizer and adjusting the embedding layer. 

The adaptation procedure is explained in [this blog post](https://medium.com/@mpamies247/ee1ebc70bc79).

## Training

### Training data

The training corpus consists of 26B tokens of several corpora gathered from web crawlings and public domain data.

| Dataset             | Language | Words (per-epoch) | Epochs       |
|---------------------|----------|--------------------|--------------|
| Wikipedia           | en       |           2169.97M |  1.428144485 |
| C4_es               | es       |          53709.80M | 0.1049686196 |
| Biomedical          | es       |            455.03M | 0.7140722425 |
| Legal               | es       |            995.70M | 0.7140722425 |
| Wikipedia           | es       |            693.60M |  1.428144485 |
| Gutenberg           | es       |             53.18M | 0.7140722425 |
| C4_ca               | ca       |           2826.00M |  2.142216727 |
| Biomedical          | ca       |             11.80M |  1.428144485 |
| RacoCatalà Noticias | ca       |             17.16M |  2.142216727 |
| RacoCatalà Forums   | ca       |            333.73M |  2.142216727 |
| CaWaC               | ca       |             57.79M |  2.142216727 |
| Wikipedia           | ca       |            228.01M |  3.570361212 |
| Vilaweb             | ca       |             50.34M |  2.142216727 |

The dataset has the following language distribution:

|Language|Percentage|
|--------|----------|
|   En   |  16.84%  |
|   Es   |  41.38%  |
|   Ca   |  41.79%  |

Note: A small amount of English data was kept to avoid catastrophic forgetting. 

## Training procedure

The training corpus has been tokenized using a byte version of [Byte-Pair Encoding (BPE)](https://github.com/openai/gpt-2) with a vocabulary size of 50,257 tokens. 
After training a new tokenizer and adapting [falcon-7b](https://huggingface.co/tiiuae/falcon-7b)'s embedding layer, the model was
further pre-trained in three target languages: Catalan, Spanish and English.

The training lasted a total of 320 hours on 8 NVIDIA H100 GPUs with 80GB RAM.


### Training hyperparameters

- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- train_batch_size: 1
- eval_batch_size:  1
- total_train_batch_size: 8
- total_eval_batch_size:  8
- optimizer: Adam
- betas: (0.9,0.999)
- epsilon: 1e-08
- learning_rate: 5e-05
- lr_scheduler_type: linear
- num_epochs: 1.0


### Framework versions

- Pytorch 2.0.0
- Transformers 4.30.2
- Datasets 2.13.1
- Tokenizers 0.13.3

## Additional information

### Author
The Language Technologies Unit from Barcelona Supercomputing Center.

### Contact
For further information, please send an email to <langtech@bsc.es>.

### Copyright
Copyright(c) 2023 by Language Technologies Unit, Barcelona Supercomputing Center.

### License
[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)

### Funding
This work was funded by:
  - The [Departament de la Vicepresidència i de Polítiques Digitals i Territori de la Generalitat de Catalunya](https://politiquesdigitals.gencat.cat/ca/inici/index.html#googtrans(ca|en) within the framework of [Projecte AINA](https://politiquesdigitals.gencat.cat/ca/economia/catalonia-ai/aina).
  - The [Spanish State Secretariat for Digitalization and Artificial Intelligence](https://portal.mineco.gob.es/en-us/digitalizacionIA/Pages/sedia.aspx) within the framework of the [Plan de Impulso de las Tecnologías del Lenguaje](https://plantl.mineco.gob.es/Paginas/index.aspx).

### Disclaimer
{"language": ["es", "ca"], "library_name": "adapter-transformers", "tags": ["assistant"], "datasets": ["hlhdatscience/guanaco-spanish-dataset"], "pipeline_tag": "text-generation"}
adapter-transformers safetensors assistant text-generation es ca dataset:hlhdatscience/guanaco-spanish-dataset region:us