
# Dataset Card for NeuCLIR1

## Dataset Description

- **Website:** https://neuclir.github.io/
- **Repository:** https://github.com/NeuCLIR/download-collection

### Dataset Summary

This is the dataset created for TREC 2022 NeuCLIR Track. The collection designed to be similar to HC4 and a large portion of documents from HC4 are ported to this collection.
The documents are Web pages from Common Crawl in Chinese, Persian, and Russian.

### Languages

 - Chinese
 - Persian
 - Russian

## Dataset Structure

### Data Instances

| Split           | Documents |
|-----------------|----------:|
| `fas` (Persian) |      2.2M |
| `rus` (Russian) |      4.6M |
| `zho` (Chinese) |      3.2M |

### Data Fields
 - `id`: unique identifier for this document
 - `cc_file`: source file from connon crawl
 - `time`: extracted date/time from article
 - `title`: title extracted from article
 - `text`: extracted article body
 - `url`: source URL

## Dataset Usage

Using 🤗 Datasets:

```python
from datasets import load_dataset

dataset = load_dataset('neuclir/neuclir1')
dataset['fas'] # Persian documents
dataset['rus'] # Russian documents
dataset['zho'] # Chinese documents
```

task_categories:text-retrieval task_ids:document-retrieval annotations_creators:no-annotation language_creators:found multilinguality:multilingual size_categories:1M<n<10M source_datasets:extended|c4 language:fa language:ru language:zh license:odc-by region:us
{"annotations_creators": ["no-annotation"], "language_creators": ["found"], "language": ["fa", "ru", "zh"], "license": ["odc-by"], "multilinguality": ["multilingual"], "size_categories": ["1M<n<10M"], "source_datasets": ["extended|c4"], "task_categories": ["text-retrieval"], "task_ids": ["document-retrieval"], "pretty_name": "NeuCLIR1", "tags": []}
TAGS
#task_categories-text-retrieval #task_ids-document-retrieval #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-1M<n<10M #source_datasets-extended|c4 #language-Persian #language-Russian #language-Chinese #license-odc-by #region-us 

Dataset Card for NeuCLIR1
=========================


Dataset Description
-------------------


* Website: URL
* Repository: URL


### Dataset Summary


This is the dataset created for TREC 2022 NeuCLIR Track. The collection designed to be similar to HC4 and a large portion of documents from HC4 are ported to this collection.
The documents are Web pages from Common Crawl in Chinese, Persian, and Russian.


### Languages


* Chinese
* Persian
* Russian


Dataset Structure
-----------------


### Data Instances



### Data Fields


* 'id': unique identifier for this document
* 'cc\_file': source file from connon crawl
* 'time': extracted date/time from article
* 'title': title extracted from article
* 'text': extracted article body
* 'url': source URL


Dataset Usage
-------------


Using Datasets:



### Dataset Summary


This is the dataset created for TREC 2022 NeuCLIR Track. The collection designed to be similar to HC4 and a large portion of documents from HC4 are ported to this collection.
The documents are Web pages from Common Crawl in Chinese, Persian, and Russian. ### Languages


* Chinese
* Persian
* Russian


Dataset Structure
----------------- ### Data Instances ### Data Fields


* 'id': unique identifier for this document
* 'cc\_file': source file from connon crawl
* 'time': extracted date/time from article
* 'title': title extracted from article
* 'text': extracted article body
* 'url': source URL


Dataset Usage
-------------


Using Datasets:
TAGS
#task_categories-text-retrieval #task_ids-document-retrieval #annotations_creators-no-annotation #language_creators-found #multilinguality-multilingual #size_categories-1M<n<10M #source_datasets-extended|c4 #language-Persian #language-Russian #language-Chinese #license-odc-by #region-us 
 ### Dataset Summary


This is the dataset created for TREC 2022 NeuCLIR Track. The collection designed to be similar to HC4 and a large portion of documents from HC4 are ported to this collection.
The documents are Web pages from Common Crawl in Chinese, Persian, and Russian. ### Languages


* Chinese
* Persian
* Russian


Dataset Structure
----------------- ### Data Instances ### Data Fields


* 'id': unique identifier for this document
* 'cc\_file': source file from connon crawl
* 'time': extracted date/time from article
* 'title': title extracted from article
* 'text': extracted article body
* 'url': source URL


Dataset Usage
-------------


Using Datasets: