license: bsd-3-clause
language:
- vi
PhoGPT: Generative Pre-training for Vietnamese
We release a 7.5B-parameter generative model series named PhoGPT for Vietnamese, which includes the base pre-trained monolingual model PhoGPT-7B5 and its instruction-following variant PhoGPT-7B5-Instruct.
PhoGPT-7B5 is a Transformer decoder-based model that incorporates flash attention (Triton) and ALiBi for context length extrapolation. We pre-trained PhoGPT-7B5 from scratch on a 41GB pre-training corpus of Vietnamese texts. We then fine-tuned the pre-trained PhoGPT-7B5 for instruction following using a dataset of 150K Vietnamese prompt-response pairs, resulting in PhoGPT-7B5-Instruct.
We recommend users use the latest PhoGPT versions - PhoGPT-4B and PhoGPT-4B-Instruct - for better performance and efficiency.
For further information or requests, please go to PhoGPT's homepage!