Hugging Face's logo

RichardErkhov
/
instruction-pretrain_-_medicine-Llama3-8B-gguf

GGUF
Inference Endpoints
Model card Files Community
instruction-pretrain_-_medicine-Llama3-8B-gguf
File size: 10,642 Bytes 
ec7b953
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
 
Quantization made by Richard Erkhov.

[Github](https://github.com/RichardErkhov)

[Discord](https://discord.gg/pvy7H8DZMG)

[Request more models](https://github.com/RichardErkhov/quant_request)


medicine-Llama3-8B - GGUF
- Model creator: https://huggingface.co/instruction-pretrain/
- Original model: https://huggingface.co/instruction-pretrain/medicine-Llama3-8B/


| Name | Quant method | Size |
| ---- | ---- | ---- |
| [medicine-Llama3-8B.Q2_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q2_K.gguf) | Q2_K | 2.96GB |
| [medicine-Llama3-8B.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.IQ3_XS.gguf) | IQ3_XS | 3.28GB |
| [medicine-Llama3-8B.IQ3_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.IQ3_S.gguf) | IQ3_S | 3.43GB |
| [medicine-Llama3-8B.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q3_K_S.gguf) | Q3_K_S | 3.41GB |
| [medicine-Llama3-8B.IQ3_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.IQ3_M.gguf) | IQ3_M | 3.52GB |
| [medicine-Llama3-8B.Q3_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q3_K.gguf) | Q3_K | 3.74GB |
| [medicine-Llama3-8B.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q3_K_M.gguf) | Q3_K_M | 3.74GB |
| [medicine-Llama3-8B.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q3_K_L.gguf) | Q3_K_L | 4.03GB |
| [medicine-Llama3-8B.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.IQ4_XS.gguf) | IQ4_XS | 4.18GB |
| [medicine-Llama3-8B.Q4_0.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q4_0.gguf) | Q4_0 | 4.34GB |
| [medicine-Llama3-8B.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.IQ4_NL.gguf) | IQ4_NL | 4.38GB |
| [medicine-Llama3-8B.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q4_K_S.gguf) | Q4_K_S | 4.37GB |
| [medicine-Llama3-8B.Q4_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q4_K.gguf) | Q4_K | 4.58GB |
| [medicine-Llama3-8B.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q4_K_M.gguf) | Q4_K_M | 4.58GB |
| [medicine-Llama3-8B.Q4_1.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q4_1.gguf) | Q4_1 | 4.78GB |
| [medicine-Llama3-8B.Q5_0.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q5_0.gguf) | Q5_0 | 5.21GB |
| [medicine-Llama3-8B.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q5_K_S.gguf) | Q5_K_S | 5.21GB |
| [medicine-Llama3-8B.Q5_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q5_K.gguf) | Q5_K | 5.34GB |
| [medicine-Llama3-8B.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q5_K_M.gguf) | Q5_K_M | 5.34GB |
| [medicine-Llama3-8B.Q5_1.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q5_1.gguf) | Q5_1 | 5.65GB |
| [medicine-Llama3-8B.Q6_K.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q6_K.gguf) | Q6_K | 6.14GB |
| [medicine-Llama3-8B.Q8_0.gguf](https://huggingface.co/RichardErkhov/instruction-pretrain_-_medicine-Llama3-8B-gguf/blob/main/medicine-Llama3-8B.Q8_0.gguf) | Q8_0 | 7.95GB |




Original model description:
---
datasets:
- instruction-pretrain/medicine-instruction-augmented-corpora
- Open-Orca/OpenOrca
- EleutherAI/pile
- GAIR/lima
- WizardLM/WizardLM_evol_instruct_V2_196k
language:
- en
license: llama3
tags:
- biology
- medical
---
# Instruction Pre-Training: Language Models are Supervised Multitask Learners
This repo contains the **biomedicine model developed from Llama3-8B** in our paper [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://huggingface.co/papers/2406.14491).

We explore supervised multitask pre-training by proposing ***Instruction Pre-Training***, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. ***Instruction Pre-Training* outperforms *Vanilla Pre-training* in both general pre-training from scratch and domain-adaptive continual pre-training.** In pre-training from scratch, *Instruction Pre-Training* not only improves pre-trained base models but also benefits more from further instruction tuning. **In continual pre-training, *Instruction Pre-Training* enables Llama3-8B to be comparable to or even outperform Llama3-70B.**

<p align='center'>
    <img src="/static-proxy?url=https%3A%2F%2Fcdn-uploads.huggingface.co%2Fproduction%2Fuploads%2F66711d2ee12fa6cc5f5dfc89%2FvRdsFIVQptbNaGiZ18Lih.png%26quot%3B%3C%2Fspan%3E width="400">
</p>

**************************** **Updates** ****************************
* 2024/7/31: Updated pre-training suggestions in the `Advanced Usage` section of [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
* 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M! Below, we show the performance trend on downstream tasks throughout the pre-training process:
<p align='left'>
    <img src="/static-proxy?url=https%3A%2F%2Fcdn-uploads.huggingface.co%2Fproduction%2Fuploads%2F66711d2ee12fa6cc5f5dfc89%2F0okCfRkC6uALTfuNxt0Fa.png%26quot%3B%3C%2Fspan%3E width="500">
</p>
* 2024/6/21: Released the [paper](https://huggingface.co/papers/2406.14491), [code](https://github.com/microsoft/LMOps), and [resources](https://huggingface.co/instruction-pretrain)

## Resources
**🤗 We share our data and models with example usages, feel free to open any discussions at [this page](https://huggingface.co/papers/2406.14491)! 🤗**

- Thanks to the demo [davanstrien/instruction-synthesizer](https://huggingface.co/spaces/davanstrien/instruction-synthesizer) for implementing our approach
- Context-Based Instruction Synthesizer: [instruction-synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer)
- Fine-Tuning Data for the Synthesizer: [ft-instruction-synthesizer-collection](https://huggingface.co/datasets/instruction-pretrain/ft-instruction-synthesizer-collection)
- General Models Pre-Trained from Scratch (on 100B tokes):
  - [InstructLM-500M](https://huggingface.co/instruction-pretrain/InstructLM-500M)
  - [InstructLM-1.3B](https://huggingface.co/instruction-pretrain/InstructLM-1.3B)
- Domain-Specific Models Pre-Trained from Llama3-8B:
  - [Finance-Llama3-8B](https://huggingface.co/instruction-pretrain/finance-Llama3-8B)
  - [Biomedicine-Llama3-8B](https://huggingface.co/instruction-pretrain/medicine-Llama3-8B)
- General Instruction-Augmented Corpora: [general-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/general-instruction-augmented-corpora)
- Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): [medicine-instruction-augmented-corpora](https://huggingface.co/datasets/instruction-pretrain/medicine-instruction-augmented-corpora)

## Domain-Adaptive Continued Pre-Training
Following [AdaptLLM](https://huggingface.co/AdaptLLM/medicine-chat), we augment the domain-specific raw corpora with instruction-response pairs generated by our [context-based instruction synthesizer](https://huggingface.co/instruction-pretrain/instruction-synthesizer).

### 1. To chat with the biomedicine-Llama3-8B model:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/medicine-Llama3-8B")
tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/medicine-Llama3-8B")

# Put your input here, NO prompt template is required
user_input = '''Question: Which of the following is an example of monosomy?
Options:
- 46,XX
- 47,XXX
- 69,XYY
- 45,X

Please provide your choice first and then provide explanations if possible.'''

inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device)
outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0]

answer_start = int(inputs.shape[-1])
pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True)

print(pred)
```

### 2. To evaluate our models on the domain-specific tasks
1. Setup dependencies
```bash
git clone https://github.com/microsoft/LMOps
cd LMOps/adaptllm
pip install -r requirements.txt
```

2. Evaluate
```bash
DOMAIN='biomedicine'

# if the model can fit on a single GPU: set MODEL_PARALLEL=False
# elif the model is too large to fit on a single GPU: set MODEL_PARALLEL=True
MODEL_PARALLEL=False

# number of GPUs, chosen from [1,2,4,8]
N_GPU=1

# Set as True
add_bos_token=True

bash scripts/inference.sh ${DOMAIN} 'instruction-pretrain/medicine-Llama3-8B' ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}
```


## Citation
If you find our work helpful, please cite us:

Instruction Pre-Training
```bibtex
@article{cheng2024instruction,
  title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
  author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu},
  journal={arXiv preprint arXiv:2406.14491},
  year={2024}
}
```

[Adapt LLM to Domains](https://huggingface.co/papers/2309.09530)
```bibtex
@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}
```