# GEM References

## What is it?

This repository contains all the reference datasets that are used for running evaluation on the GEM benchmark. Some of these datasets were originally hosted as a [GitHub release](https://github.com/GEM-benchmark/GEM-metrics/releases) on the [`GEM-metrics`](https://github.com/GEM-benchmark/GEM-metrics) repository, but have been migrated to the Hugging Face Hub.

## Converting datasets to JSON

We provide a `convert_dataset_to_json.py` conversion script that converts the datasets in the GEM organisation to the JSON format expected by the `GEM-metrics` library. To run the script, first install [`jq`](https://stedolan.github.io/jq/download/) and then install the script's Python dependencies:

```
python -m pip install -r requirements.txt
```

You can then run the script as follows:

```python
python generate_evaluation_datasets.py
```

This script will:

* Download and convert the datasets under the GEM organisation to JSON format
* Validate that the each dataset has the expected columns of `gem_id`, `target`, and `references`


region:us
{}
TAGS
#region-us 

# GEM References

## What is it?

This repository contains all the reference datasets that are used for running evaluation on the GEM benchmark. Some of these datasets were originally hosted as a GitHub release on the 'GEM-metrics' repository, but have been migrated to the Hugging Face Hub.

## Converting datasets to JSON

We provide a 'convert_dataset_to_json.py' conversion script that converts the datasets in the GEM organisation to the JSON format expected by the 'GEM-metrics' library. To run the script, first install 'jq' and then install the script's Python dependencies:



You can then run the script as follows:



This script will:

* Download and convert the datasets under the GEM organisation to JSON format
* Validate that the each dataset has the expected columns of 'gem_id', 'target', and 'references'


# GEM References ## What is it?

This repository contains all the reference datasets that are used for running evaluation on the GEM benchmark. Some of these datasets were originally hosted as a GitHub release on the 'GEM-metrics' repository, but have been migrated to the Hugging Face Hub. ## Converting datasets to JSON

We provide a 'convert_dataset_to_json.py' conversion script that converts the datasets in the GEM organisation to the JSON format expected by the 'GEM-metrics' library. To run the script, first install 'jq' and then install the script's Python dependencies:



You can then run the script as follows:



This script will:

* Download and convert the datasets under the GEM organisation to JSON format
* Validate that the each dataset has the expected columns of 'gem_id', 'target', and 'references'
TAGS
#region-us 
 # GEM References ## What is it?

This repository contains all the reference datasets that are used for running evaluation on the GEM benchmark. Some of these datasets were originally hosted as a GitHub release on the 'GEM-metrics' repository, but have been migrated to the Hugging Face Hub. ## Converting datasets to JSON

We provide a 'convert_dataset_to_json.py' conversion script that converts the datasets in the GEM organisation to the JSON format expected by the 'GEM-metrics' library. To run the script, first install 'jq' and then install the script's Python dependencies:



You can then run the script as follows:



This script will:

* Download and convert the datasets under the GEM organisation to JSON format
* Validate that the each dataset has the expected columns of 'gem_id', 'target', and 'references'