Hugging Face's logo

jondurbin
/
airoboros-gpt-3.5-turbo-100k-7b

Text Generation
Transformers
PyTorch
llama
text-generation-inference
Inference Endpoints
Model card Files Community
4
airoboros-gpt-3.5-turbo-100k-7b / README.md
jondurbin's picture
jondurbin
Update README.md
b1728be
preview code
|
raw
history blame
6.88 kB
metadata
license: other

airoboros-gpt-3.5-turbo-100k-7b

This is a 7b parameter, fine-tuned on 100k synthetic instruction/response pairs generated by gpt-3.5-turbo using (airoboros)[https://github.com/jondurbin/airoboros]

Links:

Prompt generation 

airoboros generate-instructions --instruction-count 100000 --concurrency 100 --temperature 1.0

Fine-tuning

The instructions.jsonl file was converted to conversation style expected by the FastChat training scripts, and then trained with:

torchrun --nproc_per_node=8 --master_port=20001 train_mem.py \
        --model_name_or_path /workspace/llama-7b-hf \
        --data_path ./as_conversations.json \
        --bf16 True \
        --output_dir /workspace/airoboros-gpt-3.5-100k-7b \
        --num_train_epochs 3 \
        --per_device_train_batch_size 4 \
        --per_device_eval_batch_size 32 \
        --gradient_accumulation_steps 4 \
        --evaluation_strategy "steps" \
        --eval_steps 1500 \
        --save_strategy "steps" \
        --save_steps 1500 \
        --save_total_limit 8 \
        --learning_rate 2e-5 \
        --weight_decay 0. \
        --warmup_ratio 0.04 \
        --lr_scheduler_type "cosine" \
        --logging_steps 1 \
        --fsdp "full_shard auto_wrap offload" \
        --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
        --tf32 True \
        --model_max_length 2048 \
        --gradient_checkpointing True \
        --lazy_preprocess True

Training took roughly 22 hours on 8x nvidia A100 80GB.

Conversion to conversation style:

import json
import uuid
inputs = [json.loads(line) for line in open("instructions.jsonl").readlines()]
conversations = []
for row in inputs:
    inputs = row['instruction']
    conversations.append({
        "id": str(uuid.uuid4()),
        "conversations": [
            {
                "from": "human",
                "value": inputs,
            },
            {
                "from": "gpt",
                "value": row['response']
            },
        ],
    })
with open("as_conversations.json", "w") as outfile:
    outfile.write(json.dumps(conversations, indent=2)

Evaluation

I used the same questions from (WizardVicunaLM)[]:

instruction gpt3.5 wizard-vicuna-13b vicuna-13b wizard-7b airoboros-gpt-3.5-turbo-100k-7b
"Write a compelling product launch announcement email to inform our customers of our new software solution." 95 92 89 90 91
"Draft an apology email to a customer who experienced a delay in their order, and provide reassurance that the issue has been resolved." 94 96 90 89 91
"As a pirate captain, what would you say to your crew to motivate them to search for hidden treasure?" 95 90 80 70 85
"Imagine you are a time traveler from the year 3000. What technological advancements would you tell people about?" 95 92 90 88 85
"As a space colonist on Mars, describe your daily life and the challenges you face living on another planet." 95 90 87 85 88
"How can you assess the credibility of a source of information, such as a news article or blog post, without relying solely on the reputation of the author or publisher?" 93 85 89 87 90
"How can observing the behavior of other people in a social situation provide clues about cultural norms and expectations?" 95 90 85 92 80
"How many text messages are sent globally in a minute? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." 90 70 65 80 85
"What are the main differences between Python and JavaScript programming languages?" 90 85 80 88 82
"What are the differences between plant-based and animal-based protein sources?" 85 92 90 80 94
"Describe a scenario where artificial intelligence could be used to improve the quality and efficiency of healthcare delivery." 95 90 92 89 91
"How do cultural, social, and economic factors influence people's food choices, and how can this knowledge be used to promote healthier diets?" 90 85 87 83 84
"How many words are spoken daily on Earth? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." 90 70 80 75 65
"How many lightning strikes occur on Earth each day? Try to explain your answer. Your explanation should take the reader through your reasoning step-by-step." 90 80 60 70 85

If we use gpt-3.5 as the baseline (as wizardvicuna/vicuna did), we get the following scores:

gpt3.5 wizard-vicuna-13b vicuna-13b wizard-7b airoboros-gpt-3.5-turbo-100k-7b
1.0 0.968421052631579 0.9368421052631579 0.9473684210526315 0.9578947368421052
1.0 1.0212765957446808 0.9574468085106383 0.9468085106382979 0.9680851063829787
1.0 0.9473684210526315 0.8421052631578947 0.7368421052631579 0.8947368421052632
1.0 0.968421052631579 0.9473684210526315 0.9263157894736842 0.8947368421052632
1.0 0.9473684210526315 0.9157894736842105 0.8947368421052632 0.9263157894736842
1.0 0.9139784946236559 0.956989247311828 0.9354838709677419 0.967741935483871
1.0 0.9473684210526315 0.8947368421052632 0.968421052631579 0.8421052631578947
1.0 0.7777777777777778 0.7222222222222222 0.8888888888888888 0.9444444444444444
1.0 0.9444444444444444 0.8888888888888888 0.9777777777777777 0.9111111111111111
1.0 1.0823529411764705 1.0588235294117647 0.9411764705882353 1.1058823529411765
1.0 0.9473684210526315 0.968421052631579 0.9368421052631579 0.9578947368421052
1.0 0.9444444444444444 0.9666666666666667 0.9222222222222223 0.9333333333333333
1.0 0.7777777777777778 0.8888888888888888 0.8333333333333334 0.7222222222222222
1.0 0.8888888888888888 0.6666666666666666 0.7777777777777778 0.9444444444444444

Average scores:

gpt3.5                             1.000000
wizard-vicuna-13b                  0.934090
vicuna-13b                         0.900847
wizard-7b                          0.902428
airoboros-gpt-3.5-turbo-100k-7b    0.926496

As you can see, the 7b airoboros model performs well, even compared to 13b models.

License

The model is licensed under the LLaMA model, and the dataset is licensed under the terms of OpenAI because it uses ChatGPT. Everything else is free.