text
As the amount of datasets for fine tuning chat models has grown, there's been a plethora of dataset formats emerge. The most popular of these include the formats used by Alpaca, ShareGPT and Open Assistant datasets. The datasets and their formats have also evolved from single-turn conversation to multi-turn. Many of these formats share similarities (and they all have the same goal), but handling the variations in formats across datasets is often a hassle, and source of potential bugs.
As the amount of datasets for fine tuning chat models has grown, there's been a plethora of dataset formats emerge. The most popular of these include the formats used by Alpaca, ShareGPT and Open Assistant datasets. The datasets and their formats have also evolved from single-turn conversation to multi-turn. Many of these formats share similarities (and they all have the same goal), but handling the variations in formats across datasets is often a hassle, and source of potential bugs.
new_line
new_line
text
Luckily the community seems to be converging on a simple and elegant chat dataset format: a list with each record being an array with each conversation turn being an object with a role (system, assistant or user) and content. Hugging Face uses this input format in the [Templates for Chat Models](
Luckily the community seems to be converging on a simple and elegant chat dataset format: a list with each record being an array with each conversation turn being an object with a role (system, assistant or user) and content. Hugging Face uses this input format in the [Templates for Chat Models](
link
https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates
https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates
text
) docs:
) docs:
new_line
new_line
code_fence
```python
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
```
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
python
new_line
new_line
text
Popular datasets like 
Popular datasets like 
resource
https://huggingface.co/datasets/HuggingFaceH4/no_robots
dataset
HuggingFaceH4/no_robots
https://huggingface.co/datasets/HuggingFaceH4/no_robots
text
 follow this format.
 follow this format.
new_line
new_line
text
To encourage usage of this format, I propose we give it a name: Hugging Face MessagesList format.
To encourage usage of this format, I propose we give it a name: Hugging Face MessagesList format.
new_line
new_line
text
The format is defined as:
The format is defined as:
new_line
new_line
text
- Having at least one 
- Having at least one 
inline_code
`messages`
messages
text
 column of type list.
 column of type list.
new_line
text
- Each messages record is an array containing one or more message turn objects.
- Each messages record is an array containing one or more message turn objects.
new_line
text
- A message turn must have 
- A message turn must have 
inline_code
`role`
role
text
 and 
 and 
inline_code
`content`
content
text
 keys.
 keys.
new_line
text
- 
- 
inline_code
`role`
role
text
 should be one of 
 should be one of 
inline_code
`system`
system
text
, 
, 
inline_code
`assistant`
assistant
text
 or 
 or 
inline_code
`user`
user
text
.
.
new_line
text
- 
- 
inline_code
`content`
content
text
 is the text content of the message.
 is the text content of the message.
new_line
new_line
text
This may be a small thing, but having a common dataset format will reduce wasted time data wrangling and help everyone.
This may be a small thing, but having a common dataset format will reduce wasted time data wrangling and help everyone.
As the amount of datasets for fine tuning chat models has grown, there's been a plethora of dataset formats emerge. The most popular of these include the formats used by Alpaca, ShareGPT and Open Assistant datasets. The datasets and their formats have also evolved from single-turn conversation to multi-turn. Many of these formats share similarities (and they all have the same goal), but handling the variations in formats across datasets is often a hassle, and source of potential bugs.

Luckily the community seems to be converging on a simple and elegant chat dataset format: a list with each record being an array with each conversation turn being an object with a role (system, assistant or user) and content. Hugging Face uses this input format in the [Templates for Chat Models](https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates) docs:

```python
messages = [
    {
        "role": "system",
        "content": "You are a friendly chatbot who always responds in the style of a pirate",
    },
    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
]
```

Popular datasets like https://huggingface.co/datasets/HuggingFaceH4/no_robots follow this format.

To encourage usage of this format, I propose we give it a name: Hugging Face MessagesList format.

The format is defined as:

- Having at least one `messages` column of type list.
- Each messages record is an array containing one or more message turn objects.
- A message turn must have `role` and `content` keys.
- `role` should be one of `system`, `assistant` or `user`.
- `content` is the text content of the message.

This may be a small thing, but having a common dataset format will reduce wasted time data wrangling and help everyone.
❤️
davanstrien osanseviero Weyaxi lhoestq julien-c clem merve alielfilali01 andysalerno dvilasuero samusenps afrideva IFSDrake sbrandeis
👍
victor julien-c Tonic dvilasuero TuringsSolutions madroid davanstrien clefourrier
https://cdn-avatars.huggingface.co/v1/production/uploads/5f17f0a0925b9863e28ad517/X7QKoiXbUtEZSG9jyvfk3.jpeg
Victor Mustar
victor
user
https://cdn-avatars.huggingface.co/v1/production/uploads/639afe83611f99c4be1e370a/-TtkPQJyeqwoXA-MmEO1b.jpeg
Damien Tanner
dctanner
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg
Daniel van Strien
davanstrien
user
/avatars/40bb534399cd10cdb853213edb5b7509.svg
Gokul
gokulavasan
user