license:apache-2.0 region:us
---
license: apache-2.0
---
The Amazing GPT4 powered Puffin Dataset https://huggingface.co/datasets/LDJnr/Puffin. But this dataset has been converted to the alpaca format with the input section blank.
What's special about this dataset, is that is is human to gpt4. So real examples that vary completley in behavior, great for have a LORA adapter trained with this for your models as they can help make the model behave more accuratley, and represent gpt4 outputs better.
Not a single question on this dataset is over 2048 tokens so when training a lora with this feel free to use a cutt off length of 2048

For ALPACA LORA users:
Modules you can target with lora:"gate_proj", "down_proj", "up_proj", "q_proj", "v_proj", "k_proj", "o_proj" 
Most lora models use:"q_proj", "v_proj", "k_proj", "o_proj" 
Platypus which got terrific results: "gate_proj", "down_proj", "up_proj"

Research on targeting certain modules still needs to be done, but if you don't want to train over a previously trained models newly learned abilities, target different modules than the ones used for original training.

Hyper perameters used by Platypus:
Hyperparameters for 13B and 70B Models
Hyperparameter Platypus2-13B / 70B
batch size 16
micro batch size 1
num epochs 1
learning rate 4e-4 / 3e-4
cutoff len 4096
lora rank 16
lora alpha 16
lora dropout 0.05
lora target modules gate_proj, down_proj, up_proj
train on inputs False
add eos token False
group by length False
prompt template alpaca
lr scheduler cosine
warmup steps 100

I would reccomend using a batch size of 4-10, and cutt off length to ≤ 2048 to avoid using vram issues. Load_in_4bit, Normal Float, and bf16. For single 24 gig card.
If training with oobabooga you must edit the "training.py" file in the "oobabooga_windows\text-generation-webui\modules" folder. In line 49 edit standard modules to the modules you would like to target.
If training with alpaca lora use the argument --lora_target_modules when running the train.py command. To load in 4bit you must edit the train file, adding load in 4 bit, bf16, and normal float quant.

This section was written by the datasets original creator(LDJnr):
Comprised of over 2,000 multi-turn conversations between GPT-4 and real humans.

Average context length per conversation is over 1,000 tokens. (will measure this more accurately soon)

Average turns per conversation is more than 10. (will measure this more accurately soon)

The other portion of Puffin is made of manually curated subsets of the following (All responses synthesized using GPT-4):

CamelAI/Physics CamelAI/Math CamelAI/Biology CamelAI/Chemistry

A majority of the real multi-turn conversations are made up of a curated subset of the original ShareGPT dataset.

Extensive cleaning was done to filter out instances of overt AI moralizing or related behaviour, such as "As an AI language model" and "September 2021"

Most importantly, we narrowed down the ShareGPT dataset to strictly only GPT-4 examples. Knowing which ShareGPT examples were GPT-4 vs GPT-3.5 was a task that would've been much more arduous if it wasn't for the help of folks over at OpenChat, whom annoteated the neccessary examples.

During the curation process, there can be some relatively arduos steps when it comes to actually executing on the best experimentation or concepts for how to filter examples out. Luckily there is folks over at NousResearch that helped expedite this process with little to no sacrifices in quality, big thank you to J-Supha specifically for making these types of significant contributions.

Along with J-Supha, some other people are worth mentioning, these are the folks that helped on long late night calls to help debug and/or get Puffin training on Llama-2 Asap, all within 12 hours of Llama-2 being announced.

Emozilla, Teknium, Caseus. And of course thank you to RedmondAI for sponsoring the training compute!
Future Plans & How you can help!
This is a relatively early build amongst the grand plans for the future of what I plan to work on!

In the near future we plan on leveraging the help of domain specific expert volunteers to eliminate any mathematically/verifiably incorrect answers from our training curations.

If you have at-least a bachelors in mathematics, physics, biology or chemistry and would like to volunteer even just 30 minutes of your expertise time, please contact LDJ on discord!