---
# Bigcode PII Training Dataset

## Dataset Description
This is the dataset used for the training of [bigcode-pii-model](https://huggingface.co/bigcode/bigcode-pii-model) (after training on pseudo-labeled data). 
It is a concatenation of an early version of [bigcode-pii-dataset](https://huggingface.co/datasets/bigcode/bigcode-pii-dataset) which had less samples, and [pii-for-code](https://huggingface.co/datasets/bigcode/pii-for-code-v2)
(a dataset with 400 files we annotated in a previous iteration: MORE INFO TO BE ADDED).

Files with `AMBIGUOUS` and `ID` were excluded. Each PII subtype was remaped to it supertype.

## Statistics

The dataset consists of **11878** files in 31 programming languages. More statistics and information about the original annotated dataset can be found at the dataset card of: [bigcode-pii-dataset](https://huggingface.co/datasets/bigcode/bigcode-pii-dataset). 
We provide the training and test splits we used for the training and evaluation of the [bigcode-pii-model](https://huggingface.co/bigcode/bigcode-pii-model).
Below is the distribution of PII entoties in each split.

| Entity type  | Train | Validation |
|--------------|-------|------------|
| EMAIL      | 4721  | 1742       |
| NAME       | 3847  | 1298       |
| IP_ADDRESS | 1941  | 521        |
| USERNAME   | 1320  | 346        |
| PASSWORD   | 390   | 148        |
| KEY        | 171   | 118        |

# How to use

```python
from datasets import load_dataset

ds = load_dataset("bigcode/bigcode-pii-dataset-training")
```

```
DatasetDict({
    train: Dataset({
        features: ['id', 'content', 'language', 'pii', 'assignment_id'],
        num_rows: 7878
    })
    validation: Dataset({
        features: ['id', 'content', 'language', 'pii', 'assignment_id'],
        num_rows: 4000
    })
})
```


# Considerations for Using the Data

When using this dataset, please be mindful of the data governance risks that come with handling personally identifiable information (PII). Despite sourcing the data from open, permissive GitHub repositories and having it annotated by fairly paid crowd-workers, it does contain sensitive details such as names, usernames, keys, emails, passwords, and IP addresses. To ensure responsible use for research within the open-source community, access to the dataset will be provided through a gated mechanism.

We expect researchers and developers working with the dataset to adhere to the highest ethical standards and employ robust data protection measures.
To assist users in effectively detecting and masking PII, we've also released a PII model trained on this dataset. 
Our goal in providing access to both the dataset and the PII model is to foster the development of privacy-preserving AI technologies while minimizing potential risks related to handling PII.
region:us
{"dataset_info": {"features": [{"name": "id", "dtype": "int64"}, {"name": "content", "dtype": "string"}, {"name": "language", "dtype": "string"}, {"name": "pii", "list": [{"name": "context", "dtype": "string"}, {"name": "end", "dtype": "int64"}, {"name": "start", "dtype": "int64"}, {"name": "tag", "dtype": "string"}, {"name": "value", "dtype": "string"}]}, {"name": "assignment_id", "dtype": "string"}], "splits": [{"name": "train", "num_bytes": 17215712, "num_examples": 7878}, {"name": "validation", "num_bytes": 7302111, "num_examples": 4000}], "download_size": 10754489, "dataset_size": 24517823}, "extra_gated_prompt": "## Terms of Use for the dataset\n\nThis is an annotated dataset for Personal Identifiable Information (PII) in code. We ask that you read and agree to the following Terms of Use before using the dataset and fill this [form](https://docs.google.com/forms/d/e/1FAIpQLSfiWKyBB8-PxOCLo-KMsLlYNyQNJEzxJw0gcUAUHT3UY848qA/viewform):\n1. You agree that you will not use the PII dataset for any purpose other than training or evaluating models for PII removal from datasets.\n2. You agree that you will not share the PII dataset or any modified versions for whatever purpose.\n3. Unless required by applicable law or agreed to in writing, the dataset is provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using the dataset, and assume any risks associated with your exercise of permissions under these Terms of Use.\n4. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE DATASET OR THE USE OR OTHER DEALINGS IN THE DATASET.", "extra_gated_fields": {"Email": "text", "I have read the License and agree with its terms": "checkbox"}}
TAGS
#region-us 



---


Bigcode PII Training Dataset
============================


Dataset Description
-------------------


This is the dataset used for the training of bigcode-pii-model (after training on pseudo-labeled data).
It is a concatenation of an early version of bigcode-pii-dataset which had less samples, and pii-for-code
(a dataset with 400 files we annotated in a previous iteration: MORE INFO TO BE ADDED).


Files with 'AMBIGUOUS' and 'ID' were excluded. Each PII subtype was remaped to it supertype.


Statistics
----------


The dataset consists of 11878 files in 31 programming languages. More statistics and information about the original annotated dataset can be found at the dataset card of: bigcode-pii-dataset.
We provide the training and test splits we used for the training and evaluation of the bigcode-pii-model.
Below is the distribution of PII entoties in each split.


Entity type: EMAIL, Train: 4721, Validation: 1742
Entity type: NAME, Train: 3847, Validation: 1298
Entity type: IP\_ADDRESS, Train: 1941, Validation: 521
Entity type: USERNAME, Train: 1320, Validation: 346
Entity type: PASSWORD, Train: 390, Validation: 148
Entity type: KEY, Train: 171, Validation: 118


How to use
==========


Considerations for Using the Data
=================================


When using this dataset, please be mindful of the data governance risks that come with handling personally identifiable information (PII). Despite sourcing the data from open, permissive GitHub repositories and having it annotated by fairly paid crowd-workers, it does contain sensitive details such as names, usernames, keys, emails, passwords, and IP addresses. To ensure responsible use for research within the open-source community, access to the dataset will be provided through a gated mechanism.


We expect researchers and developers working with the dataset to adhere to the highest ethical standards and employ robust data protection measures.
To assist users in effectively detecting and masking PII, we've also released a PII model trained on this dataset.
Our goal in providing access to both the dataset and the PII model is to foster the development of privacy-preserving AI technologies while minimizing potential risks related to handling PII.



TAGS
#region-us 

passage: TAGS
#region-us 
