# Dataset Card for citizen_nlu

## Table of Contents

- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

### Dataset Description

- **Homepage**: [NeuralSpace Homepage](https://huggingface.co/neuralspace)
- **Repository:** [citizen_nlu Dataset](https://huggingface.co/datasets/neuralspace/citizen_nlu)
- **Point of Contact:** [Juhi Jain](mailto:juhi@neuralspace.ai)
- **Point of Contact:** [Ayushman Dash](mailto:ayushman@neuralspace.ai)
- **Size of downloaded dataset files:** 67.6 MB

### Dataset Summary

NeuralSpace strives to provide AutoNLP text and speech services, especially for low-resource languages. One of the major services provided by NeuralSpace on its platform is the “Language Understanding” service, where you can build, train and deploy your NLU model to recognize intents and entities with minimal code and just a few clicks.

The initiative of this challenge is created with the purpose of sparkling AI applications to address some of the pressing problems in India and find unique ways to address them. Starting with a focus on NLU, this challenge hopes to make progress towards multilingual modelling, as language diversity is significantly underserved on the web.

NeuralSpace aims at mastering the low-resource domain, and the citizen services use case is naturally a multilingual and essential domain for the general citizen.

Citizen services refer to the essential services provided by organizations to general citizens. In this case, we focus on important services like various FIR-based requests, Blood/Platelets Donation, and Coronavirus-related queries. 

Such services may not be needed regularly by any particular city but when needed are of utmost importance, and in general, the needs for such services are prevalent every day.

Despite the importance of citizen services, linguistically rich countries like India are still far behind in delivering such essential needs to the citizens with absolute ease. The best services currently available do not exist in various low-resource languages that are native to different groups of people. This challenge aims to make government services more efficient, responsive, and customer-friendly.

As our computing resources and modelling capabilities grow, so does our potential to support our citizens by delivering a far superior customer experience. Equipping a Citizen services bot with the ability to converse in vernacular languages would make them accessible to a vast group of people for whom English is not a language of choice, but for who are increasingly turning to digital platforms and interfaces for a wide range of needs and wants.

### Supported Tasks

A key component of any chatbot system is the NLU pipeline for ‘Intent Classification’ and ‘Named Entity Recognition. This primarily enables any chatbot to perform various tasks at ease. A fully functional multilingual chatbot needs to be able to decipher the language and understand exactly what the user wants. 


#### citizen_nlu

A manually-curated multilingual dataset by Data Engineers at [NeuralSpace](https://www.neuralspace.ai/) for citizen services in 9 Indian languages for a realistic information-seeking task with data samples written by native-speaking expert data annotators [here](https://www.neuralspace.ai/). The dataset files are available in CSV format.

### Languages

The citizen_nlu data is available in nine Indian languages i.e,  Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu

## Dataset Structure

### Data Instances

- **Size of downloaded dataset files:** 67.6 MB

An example of 'test' looks as follows.

``` text,intents
मेरे पिता की कार उनके कार्यालय की पार्किंग से  कल  से गायब है। वाहन संख्या  केए-03-एचए-1985 । मैं एफआईआर कराना चाहता हूं।,ReportingMissingVehicle
 ```

An example of 'train' looks as follows.

```text,intents
என் தாத்தா எனக்கு பிறந்தநாள் பரிசு கொடுத்தார் மஞ்சள் நான் டாடனானோவை இழந்தேன். காணவில்லை என புகார் தெரிவிக்க விரும்புகிறேன்,ReportingMissingVehicle
```

### Data Fields

The data fields are the same among all splits.

#### citizen_nlu

- `text`: a `string` feature.
- `intent`: a `string` feature.
- `type`: a classification label, with possible values including `train` or `test`.

### Data Splits

#### citizen_nlu
|    |train|test|
|----|----:|---:|
|citizen_nlu| 287832| 4752|

### Contributions
Mehar Bhatia (mehar@neuralspace.ai)
task_categories:question-answering task_categories:text-retrieval task_categories:text2text-generation task_categories:other task_categories:translation task_categories:conversational task_ids:extractive-qa task_ids:closed-domain-qa task_ids:utterance-retrieval task_ids:document-retrieval task_ids:open-book-qa task_ids:closed-book-qa annotations_creators:other language_creators:other multilinguality:multilingual size_categories:n>1K source_datasets:original language:as language:bn language:gu language:hi language:kn language:mr language:pa language:ta language:te chatbots citizen services help emergency services health reporting crime region:us
{"annotations_creators": ["other"], "language_creators": ["other"], "language": ["as", "bn", "gu", "hi", "kn", "mr", "pa", "ta", "te"], "multilinguality": ["multilingual"], "size_categories": ["n>1K"], "source_datasets": ["original"], "task_categories": ["question-answering", "text-retrieval", "text2text-generation", "other", "translation", "conversational"], "task_ids": ["extractive-qa", "closed-domain-qa", "utterance-retrieval", "document-retrieval", "closed-domain-qa", "open-book-qa", "closed-book-qa"], "paperswithcode_id": "acronym-identification", "pretty_name": "Citizen Services NLU Multilingual Dataset.", "expert-generated license": ["cc-by-nc-sa-4.0"], "tags": ["chatbots", "citizen services", "help", "emergency services", "health", "reporting crime"], "configs": ["citizen_nlu"], "train-eval-index": [{"config": "citizen_nlu", "task": "token-classification", "task_id": "entity_extraction", "splits": {"train_split": "train", "eval_split": "test"}, "col_mapping": {"sentence": "text", "label": "target"}, "metrics": [{"type": "citizen_nlu", "name": "citizen_nlu", "config": "citizen_nlu"}]}]}
TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_categories-text2text-generation #task_categories-other #task_categories-translation #task_categories-conversational #task_ids-extractive-qa #task_ids-closed-domain-qa #task_ids-utterance-retrieval #task_ids-document-retrieval #task_ids-open-book-qa #task_ids-closed-book-qa #annotations_creators-other #language_creators-other #multilinguality-multilingual #size_categories-n>1K #source_datasets-original #language-Assamese #language-Bengali #language-Gujarati #language-Hindi #language-Kannada #language-Marathi #language-Panjabi #language-Tamil #language-Telugu #chatbots #citizen services #help #emergency services #health #reporting crime #region-us 

Dataset Card for citizen\_nlu
=============================


Table of Contents
-----------------


* Dataset Description
	+ Dataset Summary
	+ Supported Tasks
	+ Languages
* Dataset Structure
	+ Data Instances
	+ Data Fields
	+ Data Splits
* Dataset Creation
	+ Curation Rationale
	+ Source Data
	+ Annotations
	+ Personal and Sensitive Information
* Considerations for Using the Data
	+ Social Impact of Dataset
	+ Discussion of Biases
	+ Other Known Limitations
* Additional Information
	+ Dataset Curators
	+ Licensing Information
	+ Citation Information
	+ Contributions


### Dataset Description


* Homepage: NeuralSpace Homepage
* Repository: citizen\_nlu Dataset
* Point of Contact: Juhi Jain
* Point of Contact: Ayushman Dash
* Size of downloaded dataset files: 67.6 MB


### Dataset Summary


NeuralSpace strives to provide AutoNLP text and speech services, especially for low-resource languages. One of the major services provided by NeuralSpace on its platform is the “Language Understanding” service, where you can build, train and deploy your NLU model to recognize intents and entities with minimal code and just a few clicks.


The initiative of this challenge is created with the purpose of sparkling AI applications to address some of the pressing problems in India and find unique ways to address them. Starting with a focus on NLU, this challenge hopes to make progress towards multilingual modelling, as language diversity is significantly underserved on the web.


NeuralSpace aims at mastering the low-resource domain, and the citizen services use case is naturally a multilingual and essential domain for the general citizen.


Citizen services refer to the essential services provided by organizations to general citizens. In this case, we focus on important services like various FIR-based requests, Blood/Platelets Donation, and Coronavirus-related queries.


Such services may not be needed regularly by any particular city but when needed are of utmost importance, and in general, the needs for such services are prevalent every day.


Despite the importance of citizen services, linguistically rich countries like India are still far behind in delivering such essential needs to the citizens with absolute ease. The best services currently available do not exist in various low-resource languages that are native to different groups of people. This challenge aims to make government services more efficient, responsive, and customer-friendly.


As our computing resources and modelling capabilities grow, so does our potential to support our citizens by delivering a far superior customer experience. Equipping a Citizen services bot with the ability to converse in vernacular languages would make them accessible to a vast group of people for whom English is not a language of choice, but for who are increasingly turning to digital platforms and interfaces for a wide range of needs and wants.


### Supported Tasks


A key component of any chatbot system is the NLU pipeline for ‘Intent Classification’ and ‘Named Entity Recognition. This primarily enables any chatbot to perform various tasks at ease. A fully functional multilingual chatbot needs to be able to decipher the language and understand exactly what the user wants.


#### citizen\_nlu


A manually-curated multilingual dataset by Data Engineers at NeuralSpace for citizen services in 9 Indian languages for a realistic information-seeking task with data samples written by native-speaking expert data annotators here. The dataset files are available in CSV format.


### Languages


The citizen\_nlu data is available in nine Indian languages i.e, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu


Dataset Structure
-----------------


### Data Instances


* Size of downloaded dataset files: 67.6 MB


An example of 'test' looks as follows.


An example of 'train' looks as follows.


### Data Fields


The data fields are the same among all splits.


#### citizen\_nlu


* 'text': a 'string' feature.
* 'intent': a 'string' feature.
* 'type': a classification label, with possible values including 'train' or 'test'.


### Data Splits


#### citizen\_nlu



### Contributions


Mehar Bhatia (mehar@URL)



### Dataset Description


* Homepage: NeuralSpace Homepage
* Repository: citizen\_nlu Dataset
* Point of Contact: Juhi Jain
* Point of Contact: Ayushman Dash
* Size of downloaded dataset files: 67.6 MB ### Dataset Summary


NeuralSpace strives to provide AutoNLP text and speech services, especially for low-resource languages. One of the major services provided by NeuralSpace on its platform is the “Language Understanding” service, where you can build, train and deploy your NLU model to recognize intents and entities with minimal code and just a few clicks.


The initiative of this challenge is created with the purpose of sparkling AI applications to address some of the pressing problems in India and find unique ways to address them. Starting with a focus on NLU, this challenge hopes to make progress towards multilingual modelling, as language diversity is significantly underserved on the web.


NeuralSpace aims at mastering the low-resource domain, and the citizen services use case is naturally a multilingual and essential domain for the general citizen.


Citizen services refer to the essential services provided by organizations to general citizens. In this case, we focus on important services like various FIR-based requests, Blood/Platelets Donation, and Coronavirus-related queries.


Such services may not be needed regularly by any particular city but when needed are of utmost importance, and in general, the needs for such services are prevalent every day.


Despite the importance of citizen services, linguistically rich countries like India are still far behind in delivering such essential needs to the citizens with absolute ease. The best services currently available do not exist in various low-resource languages that are native to different groups of people. This challenge aims to make government services more efficient, responsive, and customer-friendly.


As our computing resources and modelling capabilities grow, so does our potential to support our citizens by delivering a far superior customer experience. Equipping a Citizen services bot with the ability to converse in vernacular languages would make them accessible to a vast group of people for whom English is not a language of choice, but for who are increasingly turning to digital platforms and interfaces for a wide range of needs and wants. ### Supported Tasks


A key component of any chatbot system is the NLU pipeline for ‘Intent Classification’ and ‘Named Entity Recognition. This primarily enables any chatbot to perform various tasks at ease. A fully functional multilingual chatbot needs to be able to decipher the language and understand exactly what the user wants. #### citizen\_nlu


A manually-curated multilingual dataset by Data Engineers at NeuralSpace for citizen services in 9 Indian languages for a realistic information-seeking task with data samples written by native-speaking expert data annotators here. The dataset files are available in CSV format. ### Languages


The citizen\_nlu data is available in nine Indian languages i.e, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu


Dataset Structure
----------------- ### Data Instances


* Size of downloaded dataset files: 67.6 MB


An example of 'test' looks as follows.


An example of 'train' looks as follows. ### Data Fields


The data fields are the same among all splits. #### citizen\_nlu


* 'text': a 'string' feature.
* 'intent': a 'string' feature.
* 'type': a classification label, with possible values including 'train' or 'test'. ### Data Splits #### citizen\_nlu ### Contributions


Mehar Bhatia (mehar@URL)
TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_categories-text2text-generation #task_categories-other #task_categories-translation #task_categories-conversational #task_ids-extractive-qa #task_ids-closed-domain-qa #task_ids-utterance-retrieval #task_ids-document-retrieval #task_ids-open-book-qa #task_ids-closed-book-qa #annotations_creators-other #language_creators-other #multilinguality-multilingual #size_categories-n>1K #source_datasets-original #language-Assamese #language-Bengali #language-Gujarati #language-Hindi #language-Kannada #language-Marathi #language-Panjabi #language-Tamil #language-Telugu #chatbots #citizen services #help #emergency services #health #reporting crime #region-us 
 ### Dataset Description


* Homepage: NeuralSpace Homepage
* Repository: citizen\_nlu Dataset
* Point of Contact: Juhi Jain
* Point of Contact: Ayushman Dash
* Size of downloaded dataset files: 67.6 MB ### Dataset Summary


NeuralSpace strives to provide AutoNLP text and speech services, especially for low-resource languages. One of the major services provided by NeuralSpace on its platform is the “Language Understanding” service, where you can build, train and deploy your NLU model to recognize intents and entities with minimal code and just a few clicks.


The initiative of this challenge is created with the purpose of sparkling AI applications to address some of the pressing problems in India and find unique ways to address them. Starting with a focus on NLU, this challenge hopes to make progress towards multilingual modelling, as language diversity is significantly underserved on the web.


NeuralSpace aims at mastering the low-resource domain, and the citizen services use case is naturally a multilingual and essential domain for the general citizen.


Citizen services refer to the essential services provided by organizations to general citizens. In this case, we focus on important services like various FIR-based requests, Blood/Platelets Donation, and Coronavirus-related queries.


Such services may not be needed regularly by any particular city but when needed are of utmost importance, and in general, the needs for such services are prevalent every day.


Despite the importance of citizen services, linguistically rich countries like India are still far behind in delivering such essential needs to the citizens with absolute ease. The best services currently available do not exist in various low-resource languages that are native to different groups of people. This challenge aims to make government services more efficient, responsive, and customer-friendly.


As our computing resources and modelling capabilities grow, so does our potential to support our citizens by delivering a far superior customer experience. Equipping a Citizen services bot with the ability to converse in vernacular languages would make them accessible to a vast group of people for whom English is not a language of choice, but for who are increasingly turning to digital platforms and interfaces for a wide range of needs and wants. ### Supported Tasks


A key component of any chatbot system is the NLU pipeline for ‘Intent Classification’ and ‘Named Entity Recognition. This primarily enables any chatbot to perform various tasks at ease. A fully functional multilingual chatbot needs to be able to decipher the language and understand exactly what the user wants. #### citizen\_nlu


A manually-curated multilingual dataset by Data Engineers at NeuralSpace for citizen services in 9 Indian languages for a realistic information-seeking task with data samples written by native-speaking expert data annotators here. The dataset files are available in CSV format. ### Languages


The citizen\_nlu data is available in nine Indian languages i.e, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Punjabi, Tamil, and Telugu


Dataset Structure
----------------- ### Data Instances


* Size of downloaded dataset files: 67.6 MB


An example of 'test' looks as follows.


An example of 'train' looks as follows. ### Data Fields


The data fields are the same among all splits. #### citizen\_nlu


* 'text': a 'string' feature.
* 'intent': a 'string' feature.
* 'type': a classification label, with possible values including 'train' or 'test'. ### Data Splits #### citizen\_nlu ### Contributions


Mehar Bhatia (mehar@URL)
passage: TAGS
#task_categories-question-answering #task_categories-text-retrieval #task_categories-text2text-generation #task_categories-other #task_categories-translation #task_categories-conversational #task_ids-extractive-qa #task_ids-closed-domain-qa #task_ids-utterance-retrieval #task_ids-document-retrieval #task_ids-open-book-qa #task_ids-closed-book-qa #annotations_creators-other #language_creators-other #multilinguality-multilingual #size_categories-n>1K #source_datasets-original #language-Assamese #language-Bengali #language-Gujarati #language-Hindi #language-Kannada #language-Marathi #language-Panjabi #language-Tamil #language-Telugu #chatbots #citizen services #help #emergency services #health #reporting crime #region-us 
### Dataset Description


* Homepage: NeuralSpace Homepage
* Repository: citizen\_nlu Dataset
* Point of Contact: Juhi Jain
* Point of Contact: Ayushman Dash
* Size of downloaded dataset files: 67.6 MB