
# Dataset Card for comparison-data-falcon-with-feedback

This dataset has been created with [Argilla](https://docs.argilla.io).

As shown in the sections below, this dataset can be loaded into Argilla as explained in [Load with Argilla](#load-with-argilla), or used directly with the `datasets` library in [Load with `datasets`](#load-with-datasets).

## Dataset Description

- **Homepage:** https://argilla.io
- **Repository:** https://github.com/argilla-io/argilla
- **Paper:** 
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary

This dataset contains:

* A dataset configuration file conforming to the Argilla dataset format named `argilla.cfg`. This configuration file will be used to configure the dataset when using the `FeedbackDataset.from_huggingface` method in Argilla.

* Dataset records in a format compatible with HuggingFace `datasets`. These records will be loaded automatically when using `FeedbackDataset.from_huggingface` and can be loaded independently using the `datasets` library via `load_dataset`.

* The [annotation guidelines](#annotation-guidelines) that have been used for building and curating the dataset, if they've been defined in Argilla.

### Load with Argilla

To load with Argilla, you'll just need to install Argilla as `pip install argilla --upgrade` and then use the following code:

```python
import argilla as rg

ds = rg.FeedbackDataset.from_huggingface("argilla/comparison-data-falcon-with-feedback")
```

### Load with `datasets`

To load this dataset with `datasets`, you'll just need to install `datasets` as `pip install datasets --upgrade` and then use the following code:

```python
from datasets import load_dataset

ds = load_dataset("argilla/comparison-data-falcon-with-feedback")
```

### Supported Tasks and Leaderboards

This dataset can contain [multiple fields, questions and responses](https://docs.argilla.io/en/latest/guides/llms/conceptual_guides/data_model.html) so it can be used for different NLP tasks, depending on the configuration. The dataset structure is described in the [Dataset Structure section](#dataset-structure).

There are no leaderboards associated with this dataset.

### Languages

[More Information Needed]

## Dataset Structure

### Data in Argilla

The dataset is created in Argilla with: **fields**, **questions**, and **guidelines**.

The **fields** are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.

| Field Name | Title | Type | Required | Markdown |
| ---------- | ----- | ---- | -------- | -------- |
| instruction | User instruction | TextField | True | False |
| response-1 | Response-1 | TextField | True | False |
| response-2 | Response-2 | TextField | True | False |


The **questions** are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.

| Question Name | Title | Type | Required | Description | Values/Labels |
| ------------- | ----- | ---- | -------- | ----------- | ------------- |
| choose-best | Choose the best response: | RatingQuestion | True | Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad. | [1, 2] |


Finally, the **guidelines** are just a plain string that can be used to provide instructions to the annotators. Find those in the [annotation guidelines](#annotation-guidelines) section.

### Data Instances

An example of a dataset instance in Argilla looks as follows:

```json
{
    "external_id": "518",
    "fields": {
        "instruction": "What is Depreciation",
        "response-1": "Depreciation is the drop in value of an asset due to wear and tear, age and obsolescence (going out of date) as recorded in an organization\u0027s financial records.",
        "response-2": "What is Depreciation \u2013 10 Important Facts to Know?\nWhen a business buys a new asset, the purchase price of that asset is depreciated over time to reflect its usage and eventual obsolescence. Depreciation expense can be a tax deductible expense and is usually a non-cash expense reported on a company\u2019s income statement and balance sheet. The amount of depreciation expense a company reports each year is the difference between the original purchase price of the asset and what the current value of that asset might be. Here are 10 important facts to know about depreciation:\n1. Depreciation is a non-cash expense. It is an expense that is reported in a business\u2019s income statement and balance sheet and not a cash flow expense.\n2. Depreciation is an accounting standard and it is required to be disclosed in a business\u2019s financial statements.\n3. The amount of depreciation is usually a tax expense and not a cash expense reported on a company\u2019s income statement"
    },
    "responses": [
        {
            "status": "submitted",
            "user_id": null,
            "values": {
                "choose-best": {
                    "value": 1
                }
            }
        }
    ]
}
```

While the same record in HuggingFace `datasets` looks as follows:

```json
{
    "choose-best": {
        "status": [
            "submitted"
        ],
        "user_id": [
            null
        ],
        "value": [
            1
        ]
    },
    "external_id": "518",
    "instruction": "What is Depreciation",
    "response-1": "Depreciation is the drop in value of an asset due to wear and tear, age and obsolescence (going out of date) as recorded in an organization\u0027s financial records.",
    "response-2": "What is Depreciation \u2013 10 Important Facts to Know?\nWhen a business buys a new asset, the purchase price of that asset is depreciated over time to reflect its usage and eventual obsolescence. Depreciation expense can be a tax deductible expense and is usually a non-cash expense reported on a company\u2019s income statement and balance sheet. The amount of depreciation expense a company reports each year is the difference between the original purchase price of the asset and what the current value of that asset might be. Here are 10 important facts to know about depreciation:\n1. Depreciation is a non-cash expense. It is an expense that is reported in a business\u2019s income statement and balance sheet and not a cash flow expense.\n2. Depreciation is an accounting standard and it is required to be disclosed in a business\u2019s financial statements.\n3. The amount of depreciation is usually a tax expense and not a cash expense reported on a company\u2019s income statement"
}
```

### Data Fields

Among the dataset fields, we differentiate between the following:

* **Fields:** These are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.
    
    * **instruction** is of type `TextField`.
    * **response-1** is of type `TextField`.
    * **response-2** is of type `TextField`.

* **Questions:** These are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.
    
    * **choose-best** is of type `RatingQuestion` with the following allowed values [1, 2], and description "Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad.".

Additionally, we also have one more field which is optional and is the following:

* **external_id:** This is an optional field that can be used to provide an external ID for the dataset record. This can be useful if you want to link the dataset record to an external resource, such as a database or a file.

### Data Splits

The dataset contains a single split, which is `train`.

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation guidelines

These guidelines are based on the paper [Training Language Models to Follow Instructions with Human Feedback](https://cdn.openai.com/papers/Training_language_models_to_follow_instructions_with_human_feedback.pdf)

**You are given a text-based description of a task, submitted by a user.**

This task description may be in the form of an explicit instruction (e.g. "Write a story about a wise frog."). The task may also be specified indirectly, for example by using several examples of the desired behavior (e.g. given a sequence of movie reviews followed by their sentiment, followed by one movie review without a sentiment, you can assume the task is to predict the sentiment of the final review), or by generating the start of a desired output (e.g. given "There once was a wise frog named Julius," you can assume the task is to continue the story).

**You will also be given several text outputs, intended to help the user with their task.** Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless, and select the best according to these criteria.

In case **none of the outputs is correct, conforming to the criteria, or they're exactly equal, use the discard option to not rank them**.

For most tasks, being truthful and harmless is more important than being helpful.

By **helpful**, we mean that the output should follow the user's intention and help the user solve their task. Some examples of helpful behavior:

- Writing in clear language.
- Answering the question they meant to ask, even if they mis-asked it.
- Being sensitive to internationality (e.g., "football" shouldn't mean American football, and "the president" doesn't necessarily mean the US president).
- If an instruction is too confusing, asking for clarification and explaining why the instruction is confusing.
- Not giving overly long or rambling answers, or repeating information from the question.
- Not assuming extraneous extra context outside of what's given (besides things like facts about the world), unless that's an implied part of the task. For example, if asked to "write a polite email response to this email: email body," the output shouldn't assume "I can't make it this time, but am free next weekend." But if asked to "Write an email to Socrates," then you can safely assume the output can make up a context.

By **truthful**, we mean that the output contains accurate information and doesn't mislead the user. Some examples of truthful behavior:

- On tasks like summarization, where the output should only use information from input, not making up details that are not part of the input description.
- Not producing clearly false information about the world (e.g., making up facts or promoting conspiracies). For example, the output should not state that Hillary Clinton has served time in prison.
- Avoid generating misleading information or information with questionable authenticity. For example, if the input asks "Why did Hillary Clinton go to jail?", the output should not say "It's not totally clear," but rather should refute the premise of the question.

By **harmless**, we mean that the output should not cause physical, psychological, or social harm to people; damage to or loss of equipment or property; damage to the environment; or harm to institutions or resources necessary to human well-being. Some examples of harmless behavior:

- Treating other humans with kindness, respect, and consideration; not denigrating members of certain groups or using biased language against a particular group.
- Not generating abusive, threatening, or offensive language or promoting violence.
- Not writing sexual or violent content if it's not asked for.
- Not giving bad real-world advice or promoting illegal activity.

Evaluating model outputs may involve making trade-offs between these criteria. These trade-offs will depend on the task.

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]
size_categories:1K<n<10K rlfh argilla human-feedback region:us
{"size_categories": "1K<n<10K", "tags": ["rlfh", "argilla", "human-feedback"]}
TAGS
#size_categories-1K<n<10K #rlfh #argilla #human-feedback #region-us 

Dataset Card for comparison-data-falcon-with-feedback
=====================================================


This dataset has been created with Argilla.


As shown in the sections below, this dataset can be loaded into Argilla as explained in Load with Argilla, or used directly with the 'datasets' library in Load with 'datasets'.


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper:
* Leaderboard:
* Point of Contact:


### Dataset Summary


This dataset contains:


* A dataset configuration file conforming to the Argilla dataset format named 'URL'. This configuration file will be used to configure the dataset when using the 'FeedbackDataset.from\_huggingface' method in Argilla.
* Dataset records in a format compatible with HuggingFace 'datasets'. These records will be loaded automatically when using 'FeedbackDataset.from\_huggingface' and can be loaded independently using the 'datasets' library via 'load\_dataset'.
* The annotation guidelines that have been used for building and curating the dataset, if they've been defined in Argilla.


### Load with Argilla


To load with Argilla, you'll just need to install Argilla as 'pip install argilla --upgrade' and then use the following code:


### Load with 'datasets'


To load this dataset with 'datasets', you'll just need to install 'datasets' as 'pip install datasets --upgrade' and then use the following code:


### Supported Tasks and Leaderboards


This dataset can contain multiple fields, questions and responses so it can be used for different NLP tasks, depending on the configuration. The dataset structure is described in the Dataset Structure section.


There are no leaderboards associated with this dataset.


### Languages


Dataset Structure
-----------------


### Data in Argilla


The dataset is created in Argilla with: fields, questions, and guidelines.


The fields are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.



The questions are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.



Finally, the guidelines are just a plain string that can be used to provide instructions to the annotators. Find those in the annotation guidelines section.


### Data Instances


An example of a dataset instance in Argilla looks as follows:


While the same record in HuggingFace 'datasets' looks as follows:


### Data Fields


Among the dataset fields, we differentiate between the following:


* Fields: These are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.


	+ instruction is of type 'TextField'.
	+ response-1 is of type 'TextField'.
	+ response-2 is of type 'TextField'.
* Questions: These are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.


	+ choose-best is of type 'RatingQuestion' with the following allowed values [1, 2], and description "Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad.".


Additionally, we also have one more field which is optional and is the following:


* external\_id: This is an optional field that can be used to provide an external ID for the dataset record. This can be useful if you want to link the dataset record to an external resource, such as a database or a file.


### Data Splits


The dataset contains a single split, which is 'train'.


Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation guidelines


These guidelines are based on the paper Training Language Models to Follow Instructions with Human Feedback


You are given a text-based description of a task, submitted by a user.


This task description may be in the form of an explicit instruction (e.g. "Write a story about a wise frog."). The task may also be specified indirectly, for example by using several examples of the desired behavior (e.g. given a sequence of movie reviews followed by their sentiment, followed by one movie review without a sentiment, you can assume the task is to predict the sentiment of the final review), or by generating the start of a desired output (e.g. given "There once was a wise frog named Julius," you can assume the task is to continue the story).


You will also be given several text outputs, intended to help the user with their task. Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless, and select the best according to these criteria.


In case none of the outputs is correct, conforming to the criteria, or they're exactly equal, use the discard option to not rank them.


For most tasks, being truthful and harmless is more important than being helpful.


By helpful, we mean that the output should follow the user's intention and help the user solve their task. Some examples of helpful behavior:


* Writing in clear language.
* Answering the question they meant to ask, even if they mis-asked it.
* Being sensitive to internationality (e.g., "football" shouldn't mean American football, and "the president" doesn't necessarily mean the US president).
* If an instruction is too confusing, asking for clarification and explaining why the instruction is confusing.
* Not giving overly long or rambling answers, or repeating information from the question.
* Not assuming extraneous extra context outside of what's given (besides things like facts about the world), unless that's an implied part of the task. For example, if asked to "write a polite email response to this email: email body," the output shouldn't assume "I can't make it this time, but am free next weekend." But if asked to "Write an email to Socrates," then you can safely assume the output can make up a context.


By truthful, we mean that the output contains accurate information and doesn't mislead the user. Some examples of truthful behavior:


* On tasks like summarization, where the output should only use information from input, not making up details that are not part of the input description.
* Not producing clearly false information about the world (e.g., making up facts or promoting conspiracies). For example, the output should not state that Hillary Clinton has served time in prison.
* Avoid generating misleading information or information with questionable authenticity. For example, if the input asks "Why did Hillary Clinton go to jail?", the output should not say "It's not totally clear," but rather should refute the premise of the question.


By harmless, we mean that the output should not cause physical, psychological, or social harm to people; damage to or loss of equipment or property; damage to the environment; or harm to institutions or resources necessary to human well-being. Some examples of harmless behavior:


* Treating other humans with kindness, respect, and consideration; not denigrating members of certain groups or using biased language against a particular group.
* Not generating abusive, threatening, or offensive language or promoting violence.
* Not writing sexual or violent content if it's not asked for.
* Not giving bad real-world advice or promoting illegal activity.


Evaluating model outputs may involve making trade-offs between these criteria. These trade-offs will depend on the task.


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


### Contributions



### Dataset Summary


This dataset contains:


* A dataset configuration file conforming to the Argilla dataset format named 'URL'. This configuration file will be used to configure the dataset when using the 'FeedbackDataset.from\_huggingface' method in Argilla.
* Dataset records in a format compatible with HuggingFace 'datasets'. These records will be loaded automatically when using 'FeedbackDataset.from\_huggingface' and can be loaded independently using the 'datasets' library via 'load\_dataset'.
* The annotation guidelines that have been used for building and curating the dataset, if they've been defined in Argilla. ### Load with Argilla


To load with Argilla, you'll just need to install Argilla as 'pip install argilla --upgrade' and then use the following code: ### Load with 'datasets'


To load this dataset with 'datasets', you'll just need to install 'datasets' as 'pip install datasets --upgrade' and then use the following code: ### Supported Tasks and Leaderboards


This dataset can contain multiple fields, questions and responses so it can be used for different NLP tasks, depending on the configuration. The dataset structure is described in the Dataset Structure section.


There are no leaderboards associated with this dataset. ### Languages


Dataset Structure
----------------- ### Data in Argilla


The dataset is created in Argilla with: fields, questions, and guidelines.


The fields are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.



The questions are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.



Finally, the guidelines are just a plain string that can be used to provide instructions to the annotators. Find those in the annotation guidelines section. ### Data Instances


An example of a dataset instance in Argilla looks as follows:


While the same record in HuggingFace 'datasets' looks as follows: ### Data Fields


Among the dataset fields, we differentiate between the following:


* Fields: These are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.


	+ instruction is of type 'TextField'.
	+ response-1 is of type 'TextField'.
	+ response-2 is of type 'TextField'.
* Questions: These are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.


	+ choose-best is of type 'RatingQuestion' with the following allowed values [1, 2], and description "Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad.".


Additionally, we also have one more field which is optional and is the following:


* external\_id: This is an optional field that can be used to provide an external ID for the dataset record. This can be useful if you want to link the dataset record to an external resource, such as a database or a file. ### Data Splits


The dataset contains a single split, which is 'train'.


Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation guidelines


These guidelines are based on the paper Training Language Models to Follow Instructions with Human Feedback


You are given a text-based description of a task, submitted by a user.


This task description may be in the form of an explicit instruction (e.g. "Write a story about a wise frog."). The task may also be specified indirectly, for example by using several examples of the desired behavior (e.g. given a sequence of movie reviews followed by their sentiment, followed by one movie review without a sentiment, you can assume the task is to predict the sentiment of the final review), or by generating the start of a desired output (e.g. given "There once was a wise frog named Julius," you can assume the task is to continue the story).


You will also be given several text outputs, intended to help the user with their task. Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless, and select the best according to these criteria.


In case none of the outputs is correct, conforming to the criteria, or they're exactly equal, use the discard option to not rank them.


For most tasks, being truthful and harmless is more important than being helpful.


By helpful, we mean that the output should follow the user's intention and help the user solve their task. Some examples of helpful behavior:


* Writing in clear language.
* Answering the question they meant to ask, even if they mis-asked it.
* Being sensitive to internationality (e.g., "football" shouldn't mean American football, and "the president" doesn't necessarily mean the US president).
* If an instruction is too confusing, asking for clarification and explaining why the instruction is confusing.
* Not giving overly long or rambling answers, or repeating information from the question.
* Not assuming extraneous extra context outside of what's given (besides things like facts about the world), unless that's an implied part of the task. For example, if asked to "write a polite email response to this email: email body," the output shouldn't assume "I can't make it this time, but am free next weekend." But if asked to "Write an email to Socrates," then you can safely assume the output can make up a context.


By truthful, we mean that the output contains accurate information and doesn't mislead the user. Some examples of truthful behavior:


* On tasks like summarization, where the output should only use information from input, not making up details that are not part of the input description.
* Not producing clearly false information about the world (e.g., making up facts or promoting conspiracies). For example, the output should not state that Hillary Clinton has served time in prison.
* Avoid generating misleading information or information with questionable authenticity. For example, if the input asks "Why did Hillary Clinton go to jail?", the output should not say "It's not totally clear," but rather should refute the premise of the question.


By harmless, we mean that the output should not cause physical, psychological, or social harm to people; damage to or loss of equipment or property; damage to the environment; or harm to institutions or resources necessary to human well-being. Some examples of harmless behavior:


* Treating other humans with kindness, respect, and consideration; not denigrating members of certain groups or using biased language against a particular group.
* Not generating abusive, threatening, or offensive language or promoting violence.
* Not writing sexual or violent content if it's not asked for.
* Not giving bad real-world advice or promoting illegal activity.


Evaluating model outputs may involve making trade-offs between these criteria. These trade-offs will depend on the task. #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions
TAGS
#size_categories-1K<n<10K #rlfh #argilla #human-feedback #region-us 
 ### Dataset Summary


This dataset contains:


* A dataset configuration file conforming to the Argilla dataset format named 'URL'. This configuration file will be used to configure the dataset when using the 'FeedbackDataset.from\_huggingface' method in Argilla.
* Dataset records in a format compatible with HuggingFace 'datasets'. These records will be loaded automatically when using 'FeedbackDataset.from\_huggingface' and can be loaded independently using the 'datasets' library via 'load\_dataset'.
* The annotation guidelines that have been used for building and curating the dataset, if they've been defined in Argilla. ### Load with Argilla


To load with Argilla, you'll just need to install Argilla as 'pip install argilla --upgrade' and then use the following code: ### Load with 'datasets'


To load this dataset with 'datasets', you'll just need to install 'datasets' as 'pip install datasets --upgrade' and then use the following code: ### Supported Tasks and Leaderboards


This dataset can contain multiple fields, questions and responses so it can be used for different NLP tasks, depending on the configuration. The dataset structure is described in the Dataset Structure section.


There are no leaderboards associated with this dataset. ### Languages


Dataset Structure
----------------- ### Data in Argilla


The dataset is created in Argilla with: fields, questions, and guidelines.


The fields are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.



The questions are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.



Finally, the guidelines are just a plain string that can be used to provide instructions to the annotators. Find those in the annotation guidelines section. ### Data Instances


An example of a dataset instance in Argilla looks as follows:


While the same record in HuggingFace 'datasets' looks as follows: ### Data Fields


Among the dataset fields, we differentiate between the following:


* Fields: These are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.


	+ instruction is of type 'TextField'.
	+ response-1 is of type 'TextField'.
	+ response-2 is of type 'TextField'.
* Questions: These are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.


	+ choose-best is of type 'RatingQuestion' with the following allowed values [1, 2], and description "Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad.".


Additionally, we also have one more field which is optional and is the following:


* external\_id: This is an optional field that can be used to provide an external ID for the dataset record. This can be useful if you want to link the dataset record to an external resource, such as a database or a file. ### Data Splits


The dataset contains a single split, which is 'train'.


Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation guidelines


These guidelines are based on the paper Training Language Models to Follow Instructions with Human Feedback


You are given a text-based description of a task, submitted by a user.


This task description may be in the form of an explicit instruction (e.g. "Write a story about a wise frog."). The task may also be specified indirectly, for example by using several examples of the desired behavior (e.g. given a sequence of movie reviews followed by their sentiment, followed by one movie review without a sentiment, you can assume the task is to predict the sentiment of the final review), or by generating the start of a desired output (e.g. given "There once was a wise frog named Julius," you can assume the task is to continue the story).


You will also be given several text outputs, intended to help the user with their task. Your job is to evaluate these outputs to ensure that they are helpful, truthful, and harmless, and select the best according to these criteria.


In case none of the outputs is correct, conforming to the criteria, or they're exactly equal, use the discard option to not rank them.


For most tasks, being truthful and harmless is more important than being helpful.


By helpful, we mean that the output should follow the user's intention and help the user solve their task. Some examples of helpful behavior:


* Writing in clear language.
* Answering the question they meant to ask, even if they mis-asked it.
* Being sensitive to internationality (e.g., "football" shouldn't mean American football, and "the president" doesn't necessarily mean the US president).
* If an instruction is too confusing, asking for clarification and explaining why the instruction is confusing.
* Not giving overly long or rambling answers, or repeating information from the question.
* Not assuming extraneous extra context outside of what's given (besides things like facts about the world), unless that's an implied part of the task. For example, if asked to "write a polite email response to this email: email body," the output shouldn't assume "I can't make it this time, but am free next weekend." But if asked to "Write an email to Socrates," then you can safely assume the output can make up a context.


By truthful, we mean that the output contains accurate information and doesn't mislead the user. Some examples of truthful behavior:


* On tasks like summarization, where the output should only use information from input, not making up details that are not part of the input description.
* Not producing clearly false information about the world (e.g., making up facts or promoting conspiracies). For example, the output should not state that Hillary Clinton has served time in prison.
* Avoid generating misleading information or information with questionable authenticity. For example, if the input asks "Why did Hillary Clinton go to jail?", the output should not say "It's not totally clear," but rather should refute the premise of the question.


By harmless, we mean that the output should not cause physical, psychological, or social harm to people; damage to or loss of equipment or property; damage to the environment; or harm to institutions or resources necessary to human well-being. Some examples of harmless behavior:


* Treating other humans with kindness, respect, and consideration; not denigrating members of certain groups or using biased language against a particular group.
* Not generating abusive, threatening, or offensive language or promoting violence.
* Not writing sexual or violent content if it's not asked for.
* Not giving bad real-world advice or promoting illegal activity.


Evaluating model outputs may involve making trade-offs between these criteria. These trade-offs will depend on the task. #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions
passage: TAGS
#size_categories-1K<n<10K #rlfh #argilla #human-feedback #region-us 
### Dataset Summary


This dataset contains:


* A dataset configuration file conforming to the Argilla dataset format named 'URL'. This configuration file will be used to configure the dataset when using the 'FeedbackDataset.from\_huggingface' method in Argilla.
* Dataset records in a format compatible with HuggingFace 'datasets'. These records will be loaded automatically when using 'FeedbackDataset.from\_huggingface' and can be loaded independently using the 'datasets' library via 'load\_dataset'.
* The annotation guidelines that have been used for building and curating the dataset, if they've been defined in Argilla.### Load with Argilla


To load with Argilla, you'll just need to install Argilla as 'pip install argilla --upgrade' and then use the following code:### Load with 'datasets'


To load this dataset with 'datasets', you'll just need to install 'datasets' as 'pip install datasets --upgrade' and then use the following code:### Supported Tasks and Leaderboards


This dataset can contain multiple fields, questions and responses so it can be used for different NLP tasks, depending on the configuration. The dataset structure is described in the Dataset Structure section.


There are no leaderboards associated with this dataset.### Languages


Dataset Structure
-----------------### Data in Argilla


The dataset is created in Argilla with: fields, questions, and guidelines.


The fields are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.



The questions are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.



Finally, the guidelines are just a plain string that can be used to provide instructions to the annotators. Find those in the annotation guidelines section. passage: ### Data Instances


An example of a dataset instance in Argilla looks as follows:


While the same record in HuggingFace 'datasets' looks as follows:### Data Fields


Among the dataset fields, we differentiate between the following:


* Fields: These are the dataset records themselves, for the moment just text fields are suppported. These are the ones that will be used to provide responses to the questions.


	+ instruction is of type 'TextField'.
	+ response-1 is of type 'TextField'.
	+ response-2 is of type 'TextField'.
* Questions: These are the questions that will be asked to the annotators. They can be of different types, such as rating, text, single choice, or multiple choice.


	+ choose-best is of type 'RatingQuestion' with the following allowed values [1, 2], and description "Choose the most helpful, harmless, and truthful response. Select 1 for response-1, 2 for response-2, or discard if both are equally good/bad.".


Additionally, we also have one more field which is optional and is the following:


* external\_id: This is an optional field that can be used to provide an external ID for the dataset record. This can be useful if you want to link the dataset record to an external resource, such as a database or a file.### Data Splits


The dataset contains a single split, which is 'train'.


Dataset Creation
----------------### Curation Rationale### Source Data#### Initial Data Collection and Normalization#### Who are the source language producers?### Annotations