# Pseudo-labeled-python-data-pii-detection-filtered

This dataset was used for the training of a PII detection NER model. We annotated it using pseudo-labelelling to enhance model performance on some rare PII entities like keys.

It consists of 18,000 files annotates using an ensemble of two encoder models Deberta-v3-large and stanford-deidentifier-base which were fine-tuned on a labeled PII dataset for code with 400 files from this work. To select good-quality pseudo-labels, 
we computed the average probability logits between the models and filtered based on a minimum score. After inspection, we observed a high rate of false positives for Keys and Passwords, hence we retained only the entities that had a trigger word like key, auth and pwd in the surrounding context. 
region:us
{"dataset_info": {"features": [{"name": "lang", "dtype": "string"}, {"name": "content", "dtype": "string"}, {"name": "id", "dtype": "int64"}, {"name": "pii", "dtype": "string"}], "splits": [{"name": "filtered", "num_bytes": 221082330, "num_examples": 17678}], "download_size": 0, "dataset_size": 221082330}}
TAGS
#region-us 

# Pseudo-labeled-python-data-pii-detection-filtered

This dataset was used for the training of a PII detection NER model. We annotated it using pseudo-labelelling to enhance model performance on some rare PII entities like keys.

It consists of 18,000 files annotates using an ensemble of two encoder models Deberta-v3-large and stanford-deidentifier-base which were fine-tuned on a labeled PII dataset for code with 400 files from this work. To select good-quality pseudo-labels, 
we computed the average probability logits between the models and filtered based on a minimum score. After inspection, we observed a high rate of false positives for Keys and Passwords, hence we retained only the entities that had a trigger word like key, auth and pwd in the surrounding context. 
# Pseudo-labeled-python-data-pii-detection-filtered

This dataset was used for the training of a PII detection NER model. We annotated it using pseudo-labelelling to enhance model performance on some rare PII entities like keys.

It consists of 18,000 files annotates using an ensemble of two encoder models Deberta-v3-large and stanford-deidentifier-base which were fine-tuned on a labeled PII dataset for code with 400 files from this work. To select good-quality pseudo-labels, 
we computed the average probability logits between the models and filtered based on a minimum score. After inspection, we observed a high rate of false positives for Keys and Passwords, hence we retained only the entities that had a trigger word like key, auth and pwd in the surrounding context.
TAGS
#region-us 
 # Pseudo-labeled-python-data-pii-detection-filtered

This dataset was used for the training of a PII detection NER model. We annotated it using pseudo-labelelling to enhance model performance on some rare PII entities like keys.

It consists of 18,000 files annotates using an ensemble of two encoder models Deberta-v3-large and stanford-deidentifier-base which were fine-tuned on a labeled PII dataset for code with 400 files from this work. To select good-quality pseudo-labels, 
we computed the average probability logits between the models and filtered based on a minimum score. After inspection, we observed a high rate of false positives for Keys and Passwords, hence we retained only the entities that had a trigger word like key, auth and pwd in the surrounding context.