
# Dataset Card for XNLI Code-Mixed Corpus (Sampled)

## Dataset Description

- **Homepage:** 
- **Repository:** 
- **Paper:** 
- **Leaderboard:** 
- **Point of Contact:** 

### Dataset Summary


### Supported Tasks and Leaderboards

Binary mode classification (spoken vs written)

### Languages
- English
- German
- French
- German-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory
- French-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory

## Dataset Structure

### Data Instances

{
  'text': "And he said , Mama , I 'm home", 
  'label': 0
}

### Data Fields
- text: sentence
- label: binary label of text (0: spoken 1: written)

### Data Splits
  - monolingual
    - train (English, German, French monolingual): 2490
    - test (English, German, French monolingual): 5007
  - de_ec
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Equivalence Constraint Theory): 14543
  - de_ml
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Matrix Language Theory): 12750
  - fr_ec
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Equivalence Constraint Theory): 18653
  - fr_ml
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Matrix Language Theory): 17381

### Other Statistics


#### Average Sentence Length

- monolingual
  - train: 19.18714859437751
  - test: 19.321150389454765
 
- de_ec
  - train: 19.18714859437751
  - test: 11.24314103004882
    
- de_ml
  - train: 19.18714859437751
  - test: 12.159450980392156

- fr_ec
  - train: 19.18714859437751
  - test: 12.26526564091567
    
- fr_ml
  - train: 19.18714859437751
  - test: 13.486968528853346


#### Label Split
  - monolingual
    - train
      - 0: 498
      - 1: 1992
      
    - test
      - 0: 1002
      - 1: 4005

  - de_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2777
      - 1: 11766
  
  - de_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2329
      - 1: 10421

  - fr_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 3322
      - 1: 15331

  - fr_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2788
      - 1: 14593

## Dataset Creation

### Curation Rationale

Using the XNLI Parallel Corpus, we generated a code-mixed corpus using CodeMixed Text Generator, and sampled a maximum of 30 sentences per original English sentence.

The XNLI Parallel Corpus is available here:
https://huggingface.co/datasets/nanakonoda/xnli_parallel
It was created from the XNLI corpus.
More information is available in the datacard for the XNLI Parallel Corpus.

Here is the link and citation for the original CodeMixed Text Generator paper.
https://github.com/microsoft/CodeMixed-Text-Generator

```
@inproceedings{rizvi-etal-2021-gcm,
    title = "{GCM}: A Toolkit for Generating Synthetic Code-mixed Text",
    author = "Rizvi, Mohd Sanad Zaki  and
      Srinivasan, Anirudh  and
      Ganu, Tanuja  and
      Choudhury, Monojit  and
      Sitaram, Sunayana",
    booktitle = "Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations",
    month = apr,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.eacl-demos.24",
    pages = "205--211",
    abstract = "Code-mixing is common in multilingual communities around the world, and processing it is challenging due to the lack of labeled and unlabeled data. We describe a tool that can automatically generate code-mixed data given parallel data in two languages. We implement two linguistic theories of code-mixing, the Equivalence Constraint theory and the Matrix Language theory to generate all possible code-mixed sentences in the language-pair, followed by sampling of the generated data to generate natural code-mixed sentences. The toolkit provides three modes: a batch mode, an interactive library mode and a web-interface to address the needs of researchers, linguists and language experts. The toolkit can be used to generate unlabeled text data for pre-trained models, as well as visualize linguistic theories of code-mixing. We plan to release the toolkit as open source and extend it by adding more implementations of linguistic theories, visualization techniques and better sampling techniques. We expect that the release of this toolkit will help facilitate more research in code-mixing in diverse language pairs.",
}
```

### Source Data

XNLI Code-Mixed Corpus
https://huggingface.co/datasets/nanakonoda/xnli_cm

XNLI Parallel Corpus
https://huggingface.co/datasets/nanakonoda/xnli_parallel

#### Original Source Data
XNLI Parallel Corpus was created using the XNLI Corpus.
https://github.com/facebookresearch/XNLI

Here is the citation for the original XNLI paper.

```
@InProceedings{conneau2018xnli,
  author = "Conneau, Alexis
        and Rinott, Ruty
        and Lample, Guillaume
        and Williams, Adina
        and Bowman, Samuel R.
        and Schwenk, Holger
        and Stoyanov, Veselin",
  title = "XNLI: Evaluating Cross-lingual Sentence Representations",
  booktitle = "Proceedings of the 2018 Conference on Empirical Methods
               in Natural Language Processing",
  year = "2018",
  publisher = "Association for Computational Linguistics",
  location = "Brussels, Belgium",
}
```

#### Initial Data Collection and Normalization

We removed all punctuation from the XNLI Parallel Corpus except apostrophes.

#### Who are the source language producers?

N/A

### Annotations

#### Annotation process

N/A

#### Who are the annotators?

N/A

### Personal and Sensitive Information

N/A

## Considerations for Using the Data

### Social Impact of Dataset

N/A

### Discussion of Biases

N/A

### Other Known Limitations

N/A

## Additional Information

### Dataset Curators

N/A

### Licensing Information

N/A

### Citation Information


### Contributions

N/A
task_categories:text-classification annotations_creators:expert-generated language_creators:found multilinguality:multilingual size_categories:1M<n<10M source_datasets:extended|xnli language:en language:de language:fr mode classification aligned code-mixed region:us
{"annotations_creators": ["expert-generated"], "language_creators": ["found"], "language": ["en", "de", "fr"], "license": [], "multilinguality": ["multilingual"], "size_categories": ["1M<n<10M"], "source_datasets": ["extended|xnli"], "task_categories": ["text-classification"], "task_ids": [], "pretty_name": "XNLI Code-Mixed Corpus (Sampled)", "tags": ["mode classification", "aligned", "code-mixed"], "dataset_info": [{"config_name": "monolingual", "features": [{"name": "text", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 317164, "num_examples": 2490}, {"name": "test", "num_bytes": 641496, "num_examples": 5007}], "download_size": 891209, "dataset_size": 958660}, {"config_name": "de_ec", "features": [{"name": "text", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 317164, "num_examples": 2490}, {"name": "test", "num_bytes": 1136549, "num_examples": 14543}], "download_size": 1298619, "dataset_size": 1453713}, {"config_name": "de_ml", "features": [{"name": "text", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 317164, "num_examples": 2490}, {"name": "test", "num_bytes": 1068937, "num_examples": 12750}], "download_size": 1248962, "dataset_size": 1386101}, {"config_name": "fr_ec", "features": [{"name": "text", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 317164, "num_examples": 2490}, {"name": "test", "num_bytes": 1520429, "num_examples": 18653}], "download_size": 1644995, "dataset_size": 1837593}, {"config_name": "fr_ml", "features": [{"name": "text", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 317164, "num_examples": 2490}, {"name": "test", "num_bytes": 1544539, "num_examples": 17381}], "download_size": 1682885, "dataset_size": 1861703}], "download_size": 891209, "dataset_size": 958660}
TAGS
#task_categories-text-classification #annotations_creators-expert-generated #language_creators-found #multilinguality-multilingual #size_categories-1M<n<10M #source_datasets-extended|xnli #language-English #language-German #language-French #mode classification #aligned #code-mixed #region-us 


# Dataset Card for XNLI Code-Mixed Corpus (Sampled)

## Dataset Description

- Homepage: 
- Repository: 
- Paper: 
- Leaderboard: 
- Point of Contact: 

### Dataset Summary


### Supported Tasks and Leaderboards

Binary mode classification (spoken vs written)

### Languages
- English
- German
- French
- German-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory
- French-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory

## Dataset Structure

### Data Instances

{
  'text': "And he said , Mama , I 'm home", 
  'label': 0
}

### Data Fields
- text: sentence
- label: binary label of text (0: spoken 1: written)

### Data Splits
  - monolingual
    - train (English, German, French monolingual): 2490
    - test (English, German, French monolingual): 5007
  - de_ec
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Equivalence Constraint Theory): 14543
  - de_ml
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Matrix Language Theory): 12750
  - fr_ec
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Equivalence Constraint Theory): 18653
  - fr_ml
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Matrix Language Theory): 17381

### Other Statistics


#### Average Sentence Length

- monolingual
  - train: 19.18714859437751
  - test: 19.321150389454765
 
- de_ec
  - train: 19.18714859437751
  - test: 11.24314103004882
    
- de_ml
  - train: 19.18714859437751
  - test: 12.159450980392156

- fr_ec
  - train: 19.18714859437751
  - test: 12.26526564091567
    
- fr_ml
  - train: 19.18714859437751
  - test: 13.486968528853346


#### Label Split
  - monolingual
    - train
      - 0: 498
      - 1: 1992
      
    - test
      - 0: 1002
      - 1: 4005

  - de_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2777
      - 1: 11766
  
  - de_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2329
      - 1: 10421

  - fr_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 3322
      - 1: 15331

  - fr_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2788
      - 1: 14593

## Dataset Creation

### Curation Rationale

Using the XNLI Parallel Corpus, we generated a code-mixed corpus using CodeMixed Text Generator, and sampled a maximum of 30 sentences per original English sentence.

The XNLI Parallel Corpus is available here:
URL
It was created from the XNLI corpus.
More information is available in the datacard for the XNLI Parallel Corpus.

Here is the link and citation for the original CodeMixed Text Generator paper.
URL



### Source Data

XNLI Code-Mixed Corpus
URL

XNLI Parallel Corpus
URL

#### Original Source Data
XNLI Parallel Corpus was created using the XNLI Corpus.
URL

Here is the citation for the original XNLI paper.



#### Initial Data Collection and Normalization

We removed all punctuation from the XNLI Parallel Corpus except apostrophes.

#### Who are the source language producers?

N/A

### Annotations

#### Annotation process

N/A

#### Who are the annotators?

N/A

### Personal and Sensitive Information

N/A

## Considerations for Using the Data

### Social Impact of Dataset

N/A

### Discussion of Biases

N/A

### Other Known Limitations

N/A

## Additional Information

### Dataset Curators

N/A

### Licensing Information

N/A




### Contributions

N/A
# Dataset Card for XNLI Code-Mixed Corpus (Sampled) ## Dataset Description

- Homepage: 
- Repository: 
- Paper: 
- Leaderboard: 
- Point of Contact: ### Dataset Summary ### Supported Tasks and Leaderboards

Binary mode classification (spoken vs written) ### Languages
- English
- German
- French
- German-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory
- French-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory ## Dataset Structure ### Data Instances

{
  'text': "And he said , Mama , I 'm home", 
  'label': 0
} ### Data Fields
- text: sentence
- label: binary label of text (0: spoken 1: written) ### Data Splits
  - monolingual
    - train (English, German, French monolingual): 2490
    - test (English, German, French monolingual): 5007
  - de_ec
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Equivalence Constraint Theory): 14543
  - de_ml
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Matrix Language Theory): 12750
  - fr_ec
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Equivalence Constraint Theory): 18653
  - fr_ml
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Matrix Language Theory): 17381 ### Other Statistics #### Average Sentence Length

- monolingual
  - train: 19.18714859437751
  - test: 19.321150389454765
 
- de_ec
  - train: 19.18714859437751
  - test: 11.24314103004882
    
- de_ml
  - train: 19.18714859437751
  - test: 12.159450980392156

- fr_ec
  - train: 19.18714859437751
  - test: 12.26526564091567
    
- fr_ml
  - train: 19.18714859437751
  - test: 13.486968528853346 #### Label Split
  - monolingual
    - train
      - 0: 498
      - 1: 1992
      
    - test
      - 0: 1002
      - 1: 4005

  - de_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2777
      - 1: 11766
  
  - de_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2329
      - 1: 10421

  - fr_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 3322
      - 1: 15331

  - fr_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2788
      - 1: 14593 ## Dataset Creation ### Curation Rationale

Using the XNLI Parallel Corpus, we generated a code-mixed corpus using CodeMixed Text Generator, and sampled a maximum of 30 sentences per original English sentence.

The XNLI Parallel Corpus is available here:
URL
It was created from the XNLI corpus.
More information is available in the datacard for the XNLI Parallel Corpus.

Here is the link and citation for the original CodeMixed Text Generator paper.
URL ### Source Data

XNLI Code-Mixed Corpus
URL

XNLI Parallel Corpus
URL #### Original Source Data
XNLI Parallel Corpus was created using the XNLI Corpus.
URL

Here is the citation for the original XNLI paper. #### Initial Data Collection and Normalization

We removed all punctuation from the XNLI Parallel Corpus except apostrophes. #### Who are the source language producers?

N/A ### Annotations #### Annotation process

N/A #### Who are the annotators?

N/A ### Personal and Sensitive Information

N/A ## Considerations for Using the Data ### Social Impact of Dataset

N/A ### Discussion of Biases

N/A ### Other Known Limitations

N/A ## Additional Information ### Dataset Curators

N/A ### Licensing Information

N/A ### Contributions

N/A
TAGS
#task_categories-text-classification #annotations_creators-expert-generated #language_creators-found #multilinguality-multilingual #size_categories-1M<n<10M #source_datasets-extended|xnli #language-English #language-German #language-French #mode classification #aligned #code-mixed #region-us 
 # Dataset Card for XNLI Code-Mixed Corpus (Sampled) ## Dataset Description

- Homepage: 
- Repository: 
- Paper: 
- Leaderboard: 
- Point of Contact: ### Dataset Summary ### Supported Tasks and Leaderboards

Binary mode classification (spoken vs written) ### Languages
- English
- German
- French
- German-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory
- French-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory ## Dataset Structure ### Data Instances

{
  'text': "And he said , Mama , I 'm home", 
  'label': 0
} ### Data Fields
- text: sentence
- label: binary label of text (0: spoken 1: written) ### Data Splits
  - monolingual
    - train (English, German, French monolingual): 2490
    - test (English, German, French monolingual): 5007
  - de_ec
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Equivalence Constraint Theory): 14543
  - de_ml
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Matrix Language Theory): 12750
  - fr_ec
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Equivalence Constraint Theory): 18653
  - fr_ml
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Matrix Language Theory): 17381 ### Other Statistics #### Average Sentence Length

- monolingual
  - train: 19.18714859437751
  - test: 19.321150389454765
 
- de_ec
  - train: 19.18714859437751
  - test: 11.24314103004882
    
- de_ml
  - train: 19.18714859437751
  - test: 12.159450980392156

- fr_ec
  - train: 19.18714859437751
  - test: 12.26526564091567
    
- fr_ml
  - train: 19.18714859437751
  - test: 13.486968528853346 #### Label Split
  - monolingual
    - train
      - 0: 498
      - 1: 1992
      
    - test
      - 0: 1002
      - 1: 4005

  - de_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2777
      - 1: 11766
  
  - de_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2329
      - 1: 10421

  - fr_ec
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 3322
      - 1: 15331

  - fr_ml
    - train
      - 0: 498
      - 1: 1992
  
    - test
      - 0: 2788
      - 1: 14593 ## Dataset Creation ### Curation Rationale

Using the XNLI Parallel Corpus, we generated a code-mixed corpus using CodeMixed Text Generator, and sampled a maximum of 30 sentences per original English sentence.

The XNLI Parallel Corpus is available here:
URL
It was created from the XNLI corpus.
More information is available in the datacard for the XNLI Parallel Corpus.

Here is the link and citation for the original CodeMixed Text Generator paper.
URL ### Source Data

XNLI Code-Mixed Corpus
URL

XNLI Parallel Corpus
URL #### Original Source Data
XNLI Parallel Corpus was created using the XNLI Corpus.
URL

Here is the citation for the original XNLI paper. #### Initial Data Collection and Normalization

We removed all punctuation from the XNLI Parallel Corpus except apostrophes. #### Who are the source language producers?

N/A ### Annotations #### Annotation process

N/A #### Who are the annotators?

N/A ### Personal and Sensitive Information

N/A ## Considerations for Using the Data ### Social Impact of Dataset

N/A ### Discussion of Biases

N/A ### Other Known Limitations

N/A ## Additional Information ### Dataset Curators

N/A ### Licensing Information

N/A ### Contributions

N/A
passage: TAGS
#task_categories-text-classification #annotations_creators-expert-generated #language_creators-found #multilinguality-multilingual #size_categories-1M<n<10M #source_datasets-extended|xnli #language-English #language-German #language-French #mode classification #aligned #code-mixed #region-us 
# Dataset Card for XNLI Code-Mixed Corpus (Sampled)## Dataset Description

- Homepage: 
- Repository: 
- Paper: 
- Leaderboard: 
- Point of Contact:### Dataset Summary### Supported Tasks and Leaderboards

Binary mode classification (spoken vs written)### Languages
- English
- German
- French
- German-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory
- French-English code-mixed by Equivalence Constraint Theory
- German-English code-mixed by Matrix Language Theory## Dataset Structure### Data Instances

{
  'text': "And he said , Mama , I 'm home", 
  'label': 0
}### Data Fields
- text: sentence
- label: binary label of text (0: spoken 1: written)### Data Splits
  - monolingual
    - train (English, German, French monolingual): 2490
    - test (English, German, French monolingual): 5007
  - de_ec
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Equivalence Constraint Theory): 14543
  - de_ml
    - train (English, German, French monolingual): 2490
    - test (German-English code-mixed by Matrix Language Theory): 12750
  - fr_ec
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Equivalence Constraint Theory): 18653
  - fr_ml
    - train (English, German, French monolingual): 2490
    - test (French-English code-mixed by Matrix Language Theory): 17381### Other Statistics