
# Dataset information 
Dataset concatenating all QA datasets with context available in French and open-source.  
In addition, an augmented version of these datasets has been added (same context but different questions to create data in SQuADv2 format).  
In total, there are 221,348 training data, **910** validation data and 6,376 test data (the first 3,188 rows correspond to SQuADv2 format, the remaining 3,188 to SQuADv2 format). 
In practice, due to the restrictive license for the FQUAD 1.0 dataset, we can only share **179,886** rows of the 221,348 training data and not the test dataset.  
Our methodology is described in a blog post available in [English](https://blog.vaniila.ai/en/QA_en/) or [French](https://blog.vaniila.ai/QA/).


#  Usage
```
from datasets import load_dataset
dataset = load_dataset("CATIE-AQ/frenchQA",sep=";")
```
```
dataset
DatasetDict({
    train: Dataset({
        features: ['context', 'question', 'answer', 'answer_start', 'dataset'],
        num_rows: 179886
    })
    validation: Dataset({
        features: ['context', 'question', 'answer', 'answer_start', 'dataset'],
        num_rows: 910
    })
})
```


# Dataset
## Dataset details

| Dataset     | Format      | Train split | Dev split   | Test split  | Available in frenchQA |
| ----------- | ----------- | ----------- | ----------- | ----------- | ------------------------ |
| [piaf](https://www.data.gouv.fr/en/datasets/piaf-le-dataset-francophone-de-questions-reponses/)| SQuAD 1.0    | 9 224 Q & A  | X  | X  |  Yes |
| piaf_v2| SQuAD 2.0    | 9 224 Q & A  | X  | X  | Yes |        
| [fquad](https://fquad.illuin.tech/)| SQuAD 1.0    | 20 731 Q & A | 3 188 Q & A  (is not used for training, but as a test dataset) | 2 189 Q & A (not freely available)| No due to the license |        
| fquad_v2 | SQuAD 2.0    | 20 731 Q & A | 3 188 Q & A  (is not used for training, but as a test dataset) | X | No due to the license |         
| [lincoln/newsquadfr](https://huggingface.co/datasets/lincoln/newsquadfr) | SQuAD 1.0    | 1 650 Q & A  | 455 Q & A | X |   Yes |          
| lincoln/newsquadfr_v2 | SQuAD 2.0   | 1 650 Q & A  | 455 Q & A | X |  Yes |         
| [pragnakalp/squad_v2_french_translated](https://huggingface.co/datasets/pragnakalp/squad_v2_french_translated)| SQuAD 2.0    | 79 069 Q & A  | X  | X  |  Yes |         
| pragnakalp/squad_v2_french_translated_v2| SQuAD 2.0    | 79 069 Q & A  | X  | X  |  Yes |


## Columns
```
dataset_train = dataset['train'].to_pandas()
dataset_train.head()

   	context 	                                        question 	                                        answer 	                    answer_start 	dataset
0 	Beyoncé Giselle Knowles-Carter (/ biːˈjɒnseɪ /... 	Quand Beyonce a-t-elle commencé à devenir popu... 	à la fin des années 1990 	269 	        pragnakalp/squad_v2_french_translated
1 	Beyoncé Giselle Knowles-Carter (/ biːˈjɒnseɪ /... 	Quand Beyonce a-t-elle quitté Destiny's Child ... 	2003 	                    549 	        pragnakalp/squad_v2_french_translated
2 	Beyoncé Giselle Knowles-Carter (/ biːˈjɒnseɪ /... 	Qui a dirigé le groupe Destiny's Child ? 	        Mathew Knowles 	            376 	        pragnakalp/squad_v2_french_translated
3 	Beyoncé Giselle Knowles-Carter (/ biːˈjɒnseɪ /... 	Quand Beyoncé a-t-elle sorti Dangerously in Lo... 	2003 	                    549 	        pragnakalp/squad_v2_french_translated
4 	Beyoncé Giselle Knowles-Carter (/ biːˈjɒnseɪ /... 	Combien de Grammy Awards Beyoncé a-t-elle gagn... 	cinq 	                    629 	        pragnakalp/squad_v2_french_translated
```

- the `context` column contains the context
- the `question` column contains the question
- the `answer` column contains the answer (has been replaced by `no_answer` for rows in SQuAD v2 format)
- the `answer_start` column contains the start position of the answer in the context (has been replaced by `-1` for rows in SQuAD v2 format)
- the `dataset` column identifies the row's original dataset (if you wish to apply filters to it, rows in SQuAD v2 format are indicated with the suffix `_v2` in the dataset name)

## Split
- `train` corresponds to the concatenation of the training dataset from `pragnakalp/squad_v2_english_translated` + `lincoln/newsquadfr` + `PIAFv1.2` + the augmented version of each dataset in SQuADv2 format (no shuffle has been performed)
- `validation` corresponds to the concatenation of the newsquadfr validation dataset + this same dataset expanded in SQuAD v2 format (= newsquadfr_v2) (no shuffle performed)


# Question type statistics

The question type distribution is as follows:

| Type of question | Frequency in percent |
| -----------      | ----------- |
|What (que)        |55.02|
|Who (qui)         |15.96|
|How much (combien)|7.92|
|When (quand)      |6.90|
|Where (où)        |3.15|
|How (comment)     |3.76|
|What (quoi)       |2.60|
|Why (pourquoi)    |1.25|
|Other             |3.44|

The number of questions containing a negation, e.g. "What was the name of Chopin's first music teacher who was not an amateur musician?", is estimated at 3.55% of the total questions.


For information, the distribution of the complete dataset (containing FQUAD 1.0 and FQUAD 1.0 data in SQUAD 2.0 format) is as follows:

| Type of question | Frequency in percent |
| -----------      | ----------- |
|What (que)        |55.12|
|Who (qui)         |16.24|
|How much (combien)|7.56|
|When (quand)      |6.85|
|Where (où)        |3.98|
|How (comment)     |3.76|
|What (quoi)       |2.94|
|Why (pourquoi)    |1.41|
|Other             |2.14|

The number of questions containing a negation, e.g. "What was the name of Chopin's first music teacher who was not an amateur musician?", is estimated at 3.07% of the total questions.


# Citation
```
@misc {frenchQA2023,  
    author       = { {ALBAR, Boris and BEDU, Pierre and BOURDOIS, Loïck} },  
    organization  = { {Centre Aquitain des Technologies de l'Information et Electroniques} },  
    title        = { frenchQA (Revision 6249cd5) },  
    year         = 2023,  
    url          = { https://huggingface.co/CATIE-AQ/frenchQA },  
    doi          = { 10.57967/hf/0862 },  
    publisher    = { Hugging Face }  
}
```

# License
[cc-by-4.0](https://creativecommons.org/licenses/by/4.0/deed.en)
task_categories:question-answering size_categories:100K<n<1M language:fr license:cc-by-4.0 doi:10.57967/hf/0862 region:us
{"language": ["fr"], "license": "cc-by-4.0", "size_categories": ["100K<n<1M"], "task_categories": ["question-answering"]}
TAGS
#task_categories-question-answering #size_categories-100K<n<1M #language-French #license-cc-by-4.0 #doi-10.57967/hf/0862 #region-us 

Dataset information
===================


Dataset concatenating all QA datasets with context available in French and open-source.  

In addition, an augmented version of these datasets has been added (same context but different questions to create data in SQuADv2 format).  

In total, there are 221,348 training data, 910 validation data and 6,376 test data (the first 3,188 rows correspond to SQuADv2 format, the remaining 3,188 to SQuADv2 format).
In practice, due to the restrictive license for the FQUAD 1.0 dataset, we can only share 179,886 rows of the 221,348 training data and not the test dataset.  

Our methodology is described in a blog post available in English or French.


Usage
=====


Dataset
=======


Dataset details
---------------



Columns
-------


* the 'context' column contains the context
* the 'question' column contains the question
* the 'answer' column contains the answer (has been replaced by 'no\_answer' for rows in SQuAD v2 format)
* the 'answer\_start' column contains the start position of the answer in the context (has been replaced by '-1' for rows in SQuAD v2 format)
* the 'dataset' column identifies the row's original dataset (if you wish to apply filters to it, rows in SQuAD v2 format are indicated with the suffix '\_v2' in the dataset name)


Split
-----


* 'train' corresponds to the concatenation of the training dataset from 'pragnakalp/squad\_v2\_english\_translated' + 'lincoln/newsquadfr' + 'PIAFv1.2' + the augmented version of each dataset in SQuADv2 format (no shuffle has been performed)
* 'validation' corresponds to the concatenation of the newsquadfr validation dataset + this same dataset expanded in SQuAD v2 format (= newsquadfr\_v2) (no shuffle performed)


Question type statistics
========================


The question type distribution is as follows:



The number of questions containing a negation, e.g. "What was the name of Chopin's first music teacher who was not an amateur musician?", is estimated at 3.55% of the total questions.


For information, the distribution of the complete dataset (containing FQUAD 1.0 and FQUAD 1.0 data in SQUAD 2.0 format) is as follows:



The number of questions containing a negation, e.g. "What was the name of Chopin's first music teacher who was not an amateur musician?", is estimated at 3.07% of the total questions.


License
=======


cc-by-4.0



TAGS
#task_categories-question-answering #size_categories-100K<n<1M #language-French #license-cc-by-4.0 #doi-10.57967/hf/0862 #region-us 

passage: TAGS
#task_categories-question-answering #size_categories-100K<n<1M #language-French #license-cc-by-4.0 #doi-10.57967/hf/0862 #region-us 
