---
license: other
license_name: sspl
license_link: LICENSE
language:
- en
pipeline_tag: text-classification
tags:
- roberta
- generated_text_detection
- llm_content_detection
- AI_detection
datasets:
- Hello-SimpleAI/HC3
- tum-nlp/IDMGSP
- mlabonne/Evol-Instruct-Python-26k
library_name: transformers
---

<p style="text-align:center;">
<img src="SA_logo.png" alt="SuperAnnotate Logo" width="100" height="100"/>
</p>

<h1 align="center">SuperAnnotate</h1>
<h3 align="center">
LLM Content Detector<br/>
Fine-Tuned RoBERTa Large<br/>
</h3>

## Description

The model designed to detect generated/synthetic text. \
At the moment, such functionality is critical for check your training data and detecting fraud and cheating in scientific and educational areas.

## Model Details

### Model Description

- **Model type:** The custom architecture for binary sequence classification based on pre-trained RoBERTa, with a single output label.
- **Language(s):** Primarily English.
- **License:** Apache 2.0
- **Finetuned from model:** [RoBERTa Large](https://huggingface.co/FacebookAI/roberta-large)

### Model Sources

- **Repository:** [GitHub](https://github.com/superannotateai/generated_text_detector) for HTTP service

### Training data

The training data was sourced from three open datasets with different proportions and underwent filtering:

1. [**HC3**](https://huggingface.co/datasets/Hello-SimpleAI/HC3) | **50%**
1. [**IDMGSP**](https://huggingface.co/datasets/tum-nlp/IDMGSP) | **30%**
1. [**Evol-Instruct-Python-26k**](https://huggingface.co/datasets/mlabonne/Evol-Instruct-Python-26k) | **20%**

As a result, the training dataset contained approximately ***25k*** pairs of text-label with an approximate balance of classes. \
It's worth noting that the dataset's texts follow a logical structure: \
Human-written and model-generated texts refer to a single prompt/instruction, though the prompts themselves were not used during training.

### Peculiarity

During training, one of the priorities was not only maximizing the quality of predictions but also avoiding overfitting and obtaining an adequately confident predictor. \
We are pleased to achieve the following state of model calibration:

<img src="Calibration_plot.png" alt="SuperAnnotate Logo" width="390" height="300"/>

## Usage

TODO

## Performance

The model was evaluated on a benchmark collected from the same datasets used for training, alongside a closed subset of SuperAnnotate. \
However, there are no direct intersections of samples between the training data and the benchmark. \
The benchmark comprises 1k samples, with 200 samples per category. \
The model's performance is compared with open-source solutions and popular API detectors in the table below:

| Model/API                                                                                        | Wikipedia | Reddit QA | SA instruction | Papers |  Code  | Average |
|--------------------------------------------------------------------------------------------------|----------:|----------:|---------------:|-------:|-------:|--------:|
| [Hello-SimpleAI](https://huggingface.co/Hello-SimpleAI/chatgpt-detector-roberta)                 |   **0.97**|      0.95 |           0.82 |   0.69 |   0.47 |    0.78 |
| [RADAR](https://huggingface.co/spaces/TrustSafeAI/RADAR-AI-Text-Detector)                        |      0.47 |      0.84 |           0.59 |   0.82 |   0.65 |    0.68 |
| [GPTZero](https://gptzero.me)                                                                    |      0.72 |      0.79 |        **0.90**|   0.67 |   0.74 |    0.76 |
| [Originality.ai](https://originality.ai)                                                         |      0.91 |   **0.97**|           0.77 |**0.93**|   0.46 |    0.81 |
| [LLM content detector](https://huggingface.co/SuperAnnotate/roberta-large-llm-content-detector)  |      0.88 |      0.95 |           0.84 |   0.81 |**0.96**| **0.89**|