
## 🍮 🦙 Flan-Alpaca: Instruction Tuning from Humans and Machines

📣 Curious to know the performance of 🍮 🦙 **Flan-Alpaca** on large-scale LLM evaluation benchmark, **InstructEval**? Read our paper [https://arxiv.org/pdf/2306.04757.pdf](https://arxiv.org/pdf/2306.04757.pdf). We evaluated more than 10 open-source instruction-tuned LLMs belonging to various LLM families including Pythia, LLaMA, T5, UL2, OPT, and Mosaic. Codes and datasets: [https://github.com/declare-lab/instruct-eval](https://github.com/declare-lab/instruct-eval)


Our [repository](https://github.com/declare-lab/flan-alpaca) contains code for extending the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca)
synthetic instruction tuning to existing instruction-tuned models such as [Flan-T5](https://arxiv.org/abs/2210.11416).
The pretrained models and demos are available on HuggingFace 🤗 :

| Model                                                                     | Parameters | Training GPUs   |
|---------------------------------------------------------------------------|------------|-----------------|
| [Flan-Alpaca-Base](https://huggingface.co/declare-lab/flan-alpaca-base)   | 220M       | 1x A6000        |
| [Flan-Alpaca-Large](https://huggingface.co/declare-lab/flan-alpaca-large) | 770M       | 1x A6000        |
| [Flan-Alpaca-XL](https://huggingface.co/declare-lab/flan-alpaca-xl)       | 3B         | 1x A6000        |
| [Flan-Alpaca-XXL](https://huggingface.co/declare-lab/flan-alpaca-xxl)     | 11B        | 4x A6000 (FSDP) |

### Why?

[Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) represents an exciting new direction
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
However, the original implementation is less accessible due to licensing constraints of the
underlying [LLaMA](https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) model.
Furthermore, users have noted [potential noise](https://github.com/tloen/alpaca-lora/issues/65) in the synthetic
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
less diverse) instructions such as [Flan-T5](https://arxiv.org/abs/2210.11416).

### Usage
This uses Huggingface PEFT library for Parameter Efficient Fine Tuning

```
import torch
from peft import PeftModel
from transformers import GenerationConfig


from transformers import AutoTokenizer, AutoModelForSeq2SeqLM


BASE_MODEL = "google/flan-t5-xl"
LORA_WEIGHTS = "declare-lab/flan-alpaca-xl-lora"
TEMPERATURE = 1.0
TOP_P = 0.75
TOP_K = 40
NUM_BEAMS = 4
MAX_NEW_TOKENS = 128

if torch.cuda.is_available():
    device = "cuda"
else:
    device = "cpu"


if device == "cuda":
    model = AutoModelForSeq2SeqLM.from_pretrained(
        BASE_MODEL,
        device_map="auto",
    )
    model = PeftModel.from_pretrained(model, LORA_WEIGHTS, force_download=True)
else:
    model = AutoModelForSeq2SeqLM.from_pretrained(
        BASE_MODEL, device_map={"": device}, low_cpu_mem_usage=True
    )
    model = PeftModel.from_pretrained(
        model,
        LORA_WEIGHTS,
        device_map={"": device},
    )


prompt = "Write a short email to show that 42 is the optimal seed for training neural networks"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
input_ids = input_ids.to(device)

generation_config = GenerationConfig(
    temperature=TEMPERATURE,
    top_p=TOP_P,
    top_k=TOP_K,
    num_beams=NUM_BEAMS,
)
generation_output = model.generate(
    input_ids=input_ids,
    generation_config=generation_config,
    return_dict_in_generate=True,
    output_scores=True,
    max_new_tokens=MAX_NEW_TOKENS,
)
print(tokenizer.batch_decode(generation_output.sequences)[0])

```
{"license": "apache-2.0", "datasets": ["tatsu-lab/alpaca"]}
dataset:tatsu-lab/alpaca arxiv:2306.04757 arxiv:2210.11416 license:apache-2.0 has_space region:us
TAGS
#dataset-tatsu-lab/alpaca #arxiv-2306.04757 #arxiv-2210.11416 #license-apache-2.0 #has_space #region-us 

Flan-Alpaca: Instruction Tuning from Humans and Machines
--------------------------------------------------------


Curious to know the performance of Flan-Alpaca on large-scale LLM evaluation benchmark, InstructEval? Read our paper URL We evaluated more than 10 open-source instruction-tuned LLMs belonging to various LLM families including Pythia, LLaMA, T5, UL2, OPT, and Mosaic. Codes and datasets: URL


Our repository contains code for extending the Stanford Alpaca
synthetic instruction tuning to existing instruction-tuned models such as Flan-T5.
The pretrained models and demos are available on HuggingFace :


Model: Flan-Alpaca-Base, Parameters: 220M, Training GPUs: 1x A6000
Model: Flan-Alpaca-Large, Parameters: 770M, Training GPUs: 1x A6000
Model: Flan-Alpaca-XL, Parameters: 3B, Training GPUs: 1x A6000
Model: Flan-Alpaca-XXL, Parameters: 11B, Training GPUs: 4x A6000 (FSDP)


### Why?


Alpaca represents an exciting new direction
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
However, the original implementation is less accessible due to licensing constraints of the
underlying LLaMA model.
Furthermore, users have noted potential noise in the synthetic
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
less diverse) instructions such as Flan-T5.


### Usage


This uses Huggingface PEFT library for Parameter Efficient Fine Tuning



### Why?


Alpaca represents an exciting new direction
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
However, the original implementation is less accessible due to licensing constraints of the
underlying LLaMA model.
Furthermore, users have noted potential noise in the synthetic
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
less diverse) instructions such as Flan-T5. ### Usage


This uses Huggingface PEFT library for Parameter Efficient Fine Tuning
TAGS
#dataset-tatsu-lab/alpaca #arxiv-2306.04757 #arxiv-2210.11416 #license-apache-2.0 #has_space #region-us 
 ### Why?


Alpaca represents an exciting new direction
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
However, the original implementation is less accessible due to licensing constraints of the
underlying LLaMA model.
Furthermore, users have noted potential noise in the synthetic
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
less diverse) instructions such as Flan-T5. ### Usage


This uses Huggingface PEFT library for Parameter Efficient Fine Tuning
TAGS
#dataset-tatsu-lab/alpaca #arxiv-2306.04757 #arxiv-2210.11416 #license-apache-2.0 #has_space #region-us 
### Why?


Alpaca represents an exciting new direction
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
However, the original implementation is less accessible due to licensing constraints of the
underlying LLaMA model.
Furthermore, users have noted potential noise in the synthetic
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
less diverse) instructions such as Flan-T5.### Usage


This uses Huggingface PEFT library for Parameter Efficient Fine Tuning