
# Dataset Card for Common Crawl Domain Names
## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- **Homepage:** https://github.com/google-research-datasets/common-crawl-domain-names
- **Repository:** https://github.com/google-research-datasets/common-crawl-domain-names
- **Paper:** https://arxiv.org/pdf/2011.03138
- **Leaderboard:**
- **Point of Contact:**

### Dataset Summary

Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries (e.g. "commoncrawl" to "common crawl").

Breaking [domain names](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_is_a_URL) such as "openresearch" into component words "open" and "research" is important for applications such as Text-to-Speech synthesis and web search.  [Common Crawl](https://commoncrawl.org/)  is an open repository of web crawl data that can be accessed and analyzed by anyone. Specifically, we scraped the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so we had to manually annotate the data.

### Supported Tasks and Leaderboards

- Text-to-Speech synthesis
- Web search

### Languages

en: English

## Dataset Structure

### Data Instances

Each sample is an example of space separated segments of a domain name. The examples are stored in their original letter casing, but harder and more interesting examples can be generated by lowercasing the input first.

For example:

```
Open B S D
NASA
ASAP Workouts
```

### Data Fields

- `example`: a `string` feature: space separated segments of a domain name. 

### Data Splits

| split | size  | trivial | avg_input_length | avg_segments |
|-------|-------|---------|------------------|--------------|
| train | 17572 | 13718   | 12.63            | 2.65         |
| eval  | 1953  | 1536    | 12.77            | 2.67         |
| test  | 2170  | 1714    | 12.63            | 2.66         |

## Dataset Creation

### Curation Rationale

The dataset was curated by scraping the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so the curators of the dataset had to manually annotate the data.

### Source Data

#### Initial Data Collection and Normalization

Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

The annotators are the curators of this dataset

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

The curators of this dataset are [Jae Hun Ro](https://github.com/JaeHunRo) and [mwurts4google](https://github.com/mwurts4google), who are the contributors of the official Github repository for this dataset. Since the account handles of other curators are unknown currently, the authors of the paper linked to this dataset is mentioned here as curators, [Hao Zhang](https://arxiv.org/search/cs?searchtype=author&query=Zhang%2C+H), [Jae Ro](https://arxiv.org/search/cs?searchtype=author&query=Ro%2C+J), and [Richard Sproat](https://arxiv.org/search/cs?searchtype=author&query=Sproat%2C+R).

### Licensing Information

[MIT License](https://github.com/google-research-datasets/common-crawl-domain-names/blob/master/LICENSE)

### Citation Information

```
@inproceedings{zrs2020urlsegmentation,
  title={Semi-supervised URL Segmentation with Recurrent Neural Networks Pre-trained on Knowledge Graph Entities},
  author={Hao Zhang and Jae Ro and Richard William Sproat},
  booktitle={The 28th International Conference on Computational Linguistics (COLING 2020)},
  year={2020}
}
```


### Contributions

Thanks to [@Karthik-Bhaskar](https://github.com/Karthik-Bhaskar) for adding this dataset.
task_categories:other annotations_creators:expert-generated language_creators:crowdsourced language_creators:expert-generated language_creators:found multilinguality:monolingual size_categories:10K<n<100K source_datasets:extended|other-Common-Crawl source_datasets:original language:en license:mit web-search text-to-speech arxiv:2011.03138 region:us
{"annotations_creators": ["expert-generated"], "language_creators": ["crowdsourced", "expert-generated", "found"], "language": ["en"], "license": ["mit"], "multilinguality": ["monolingual"], "size_categories": ["10K<n<100K"], "source_datasets": ["extended|other-Common-Crawl", "original"], "task_categories": ["other"], "task_ids": [], "paperswithcode_id": "common-crawl-domain-names", "pretty_name": "Common Crawl Domain Names", "tags": ["web-search", "text-to-speech"], "dataset_info": {"features": [{"name": "example", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 321134, "num_examples": 17572}, {"name": "test", "num_bytes": 39712, "num_examples": 2170}, {"name": "validation", "num_bytes": 36018, "num_examples": 1953}], "download_size": 331763, "dataset_size": 396864}}
TAGS
#task_categories-other #annotations_creators-expert-generated #language_creators-crowdsourced #language_creators-expert-generated #language_creators-found #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-extended|other-Common-Crawl #source_datasets-original #language-English #license-mit #web-search #text-to-speech #arxiv-2011.03138 #region-us 

Dataset Card for Common Crawl Domain Names
==========================================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks and Leaderboards
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


Dataset Description
-------------------


* Homepage: URL
* Repository: URL
* Paper: URL
* Leaderboard:
* Point of Contact:


### Dataset Summary


Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries (e.g. "commoncrawl" to "common crawl").


Breaking domain names such as "openresearch" into component words "open" and "research" is important for applications such as Text-to-Speech synthesis and web search. Common Crawl is an open repository of web crawl data that can be accessed and analyzed by anyone. Specifically, we scraped the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so we had to manually annotate the data.


### Supported Tasks and Leaderboards


* Text-to-Speech synthesis
* Web search


### Languages


en: English


Dataset Structure
-----------------


### Data Instances


Each sample is an example of space separated segments of a domain name. The examples are stored in their original letter casing, but harder and more interesting examples can be generated by lowercasing the input first.


For example:


### Data Fields


* 'example': a 'string' feature: space separated segments of a domain name.


### Data Splits



Dataset Creation
----------------


### Curation Rationale


The dataset was curated by scraping the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so the curators of the dataset had to manually annotate the data.


### Source Data


#### Initial Data Collection and Normalization


Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


The annotators are the curators of this dataset


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


The curators of this dataset are Jae Hun Ro and mwurts4google, who are the contributors of the official Github repository for this dataset. Since the account handles of other curators are unknown currently, the authors of the paper linked to this dataset is mentioned here as curators, Hao Zhang, Jae Ro, and Richard Sproat.


### Licensing Information


MIT License


### Contributions


Thanks to @Karthik-Bhaskar for adding this dataset.



### Dataset Summary


Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries (e.g. "commoncrawl" to "common crawl").


Breaking domain names such as "openresearch" into component words "open" and "research" is important for applications such as Text-to-Speech synthesis and web search. Common Crawl is an open repository of web crawl data that can be accessed and analyzed by anyone. Specifically, we scraped the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so we had to manually annotate the data. ### Supported Tasks and Leaderboards


* Text-to-Speech synthesis
* Web search ### Languages


en: English


Dataset Structure
----------------- ### Data Instances


Each sample is an example of space separated segments of a domain name. The examples are stored in their original letter casing, but harder and more interesting examples can be generated by lowercasing the input first.


For example: ### Data Fields


* 'example': a 'string' feature: space separated segments of a domain name. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale


The dataset was curated by scraping the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so the curators of the dataset had to manually annotate the data. ### Source Data #### Initial Data Collection and Normalization


Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators?


The annotators are the curators of this dataset ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators


The curators of this dataset are Jae Hun Ro and mwurts4google, who are the contributors of the official Github repository for this dataset. Since the account handles of other curators are unknown currently, the authors of the paper linked to this dataset is mentioned here as curators, Hao Zhang, Jae Ro, and Richard Sproat. ### Licensing Information


MIT License ### Contributions


Thanks to @Karthik-Bhaskar for adding this dataset.
TAGS
#task_categories-other #annotations_creators-expert-generated #language_creators-crowdsourced #language_creators-expert-generated #language_creators-found #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-extended|other-Common-Crawl #source_datasets-original #language-English #license-mit #web-search #text-to-speech #arxiv-2011.03138 #region-us 
 ### Dataset Summary


Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries (e.g. "commoncrawl" to "common crawl").


Breaking domain names such as "openresearch" into component words "open" and "research" is important for applications such as Text-to-Speech synthesis and web search. Common Crawl is an open repository of web crawl data that can be accessed and analyzed by anyone. Specifically, we scraped the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so we had to manually annotate the data. ### Supported Tasks and Leaderboards


* Text-to-Speech synthesis
* Web search ### Languages


en: English


Dataset Structure
----------------- ### Data Instances


Each sample is an example of space separated segments of a domain name. The examples are stored in their original letter casing, but harder and more interesting examples can be generated by lowercasing the input first.


For example: ### Data Fields


* 'example': a 'string' feature: space separated segments of a domain name. ### Data Splits



Dataset Creation
---------------- ### Curation Rationale


The dataset was curated by scraping the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so the curators of the dataset had to manually annotate the data. ### Source Data #### Initial Data Collection and Normalization


Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators?


The annotators are the curators of this dataset ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators


The curators of this dataset are Jae Hun Ro and mwurts4google, who are the contributors of the official Github repository for this dataset. Since the account handles of other curators are unknown currently, the authors of the paper linked to this dataset is mentioned here as curators, Hao Zhang, Jae Ro, and Richard Sproat. ### Licensing Information


MIT License ### Contributions


Thanks to @Karthik-Bhaskar for adding this dataset.
passage: TAGS
#task_categories-other #annotations_creators-expert-generated #language_creators-crowdsourced #language_creators-expert-generated #language_creators-found #multilinguality-monolingual #size_categories-10K<n<100K #source_datasets-extended|other-Common-Crawl #source_datasets-original #language-English #license-mit #web-search #text-to-speech #arxiv-2011.03138 #region-us 
### Dataset Summary


Corpus of domain names scraped from Common Crawl and manually annotated to add word boundaries (e.g. "commoncrawl" to "common crawl").


Breaking domain names such as "openresearch" into component words "open" and "research" is important for applications such as Text-to-Speech synthesis and web search. Common Crawl is an open repository of web crawl data that can be accessed and analyzed by anyone. Specifically, we scraped the plaintext (WET) extracts for domain names from URLs that contained diverse letter casing (e.g. "OpenBSD"). Although in the previous example, segmentation is trivial using letter casing, this was not always the case (e.g. "NASA"), so we had to manually annotate the data.### Supported Tasks and Leaderboards


* Text-to-Speech synthesis
* Web search### Languages


en: English


Dataset Structure
-----------------### Data Instances


Each sample is an example of space separated segments of a domain name. The examples are stored in their original letter casing, but harder and more interesting examples can be generated by lowercasing the input first.


For example:### Data Fields


* 'example': a 'string' feature: space separated segments of a domain name.### Data Splits



Dataset Creation
----------------