# Dataset Card for A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era

## Dataset Description

- **Repository:** https://github.com/qwenzo/-IDMGSP
- **Paper:** TODO 

### Dataset Summary

A benchmark for detecting machine-generated scientific papers based on their abstract, introduction and conclusion sections.

### Supported Tasks and Leaderboards

current benchmark results in terms of accuracy:
| Model                       | Train Dataset   | TEST    | OOD-GPT3 | OOD-REAL | TECG    | TEST-CC |
|-----------------------------|-----------------|---------|----------|----------|---------|---------|
| LR-1gram (tf-idf) (our)    | TRAIN           | 95.3%   | 4.0%     | 94.6%    | 96.1%   | 7.8%    |
| LR-1gram (tf-idf) (our)    | TRAIN+GPT3      | 94.6%   | 86.5%    | 86.2%    | 97.8%   | 13.7%   |
| LR-1gram (tf-idf) (our)    | TRAIN-CG        | 86.6%   | 0.8%     | 97.8%    | 32.6%   | 1.2%    |
| RF-1gram (tf-idf) (our)    | TRAIN           | 94.8%   | 24.7%    | 87.3%    | 100.0%  | 8.1%    |
| RF-1gram (tf-idf) (our)    | TRAIN+GPT3      | 91.7%   | 95.0%    | 69.3%    | 100.0%  | 15.1%   |
| RF-1gram (tf-idf) (our)    | TRAIN-CG        | 97.6%   | 7.0%     | 95.0%    | 57.0%   | 1.7%    |
| [IDMGSP-Galactica-TRAIN](https://huggingface.co/tum-nlp/IDMGSP-Galactica-TRAIN) (our)        | TRAIN           | 98.4%   | 25.9%    | 95.5%    | 84.0%   | 6.8%    |
| [IDMGSP-Galactica-TRAIN_GPT3](https://huggingface.co/tum-nlp/IDMGSP-Galactica-TRAIN_GPT3) (our)            | TRAIN+GPT3      | 98.5%   | 71.2%    | 95.1%    | 84.0%   | 12.0%   |
| [IDMGSP-Galactica-TRAIN-CG](https://huggingface.co/tum-nlp/IDMGSP-Galactica-TRAIN-CG) (our)            | TRAIN-CG        | 96.4%   | 12.4%    | 97.6%    | 61.3%   | 2.4%    |
| [IDMGSP-RoBERTa-TRAIN-ABSTRACT](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-ABSTRACT) + [IDMGSP-RoBERTa-TRAIN-INTRODUCTION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-INTRODUCTION) + [IDMGSP-RoBERTa-TRAIN-CONCLUSION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CONCLUSION) (our)              | TRAIN           | 72.3%   | 55.5%    | 50.0%    | 100.0%  | 63.5%   |
| [IDMGSP-RoBERTa-TRAIN_GPT3-ABSTRACT](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN_GPT3-ABSTRACT) + [IDMGSP-RoBERTa-TRAIN_GPT3-INTRODUCTION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN_GPT3-INTRODUCTION) + [IDMGSP-RoBERTa-TRAIN_GPT3-CONCLUSION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN_GPT3-CONCLUSION) (our)              | TRAIN+GPT3      | 65.7%   | 100.0%   | 29.1%    | 100.0%  | 75.0%   |
| [IDMGSP-RoBERTa-TRAIN-CG-ABSTRACT](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CG-ABSTRACT) + [IDMGSP-RoBERTa-TRAIN-CG-INTRODUCTION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CG-INTRODUCTION) + [IDMGSP-RoBERTa-TRAIN-CG-CONCLUSION](https://huggingface.co/tum-nlp/IDMGSP-RoBERTa-TRAIN-CG-CONCLUSION) (our)              | TRAIN-CG        | 86.0%   | 2.0%     | 92.5%    | 76.5%   | 9.2%    |
| GPT-3 (our)                | TRAIN-SUB       | 100.0%  | 25.9%    | 99.0%    | 100.0%  | N/A     |
| DetectGPT                   | -               | 61.5%   | 0.0%     | 99.9%    | 68.7%   | N/A     |
| ChatGPT-IO (our)*          | -               | 69.0%   | 49.0%    | 89.0%    | 0.0%    | 3.0%    |
| LLMFE (our)*                | TRAIN+GPT3      | 80.0%   | 62.0%    | 70.0%    | 90.0%   | 33.0%   |

### Languages

English

## Dataset Structure

### Data Instances
Each instance in the dataset corresponds to a row in a CSV file, encompassing the features of a paper, its label, and the paper's source.

### Data Fields
#### classifier_input
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
  dtype: int64

#### train+gpt3
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica", "gpt3".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
  dtype: int64

#### tecg
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. Always has the value "chatgpt".
  dtype: string
- name: label
  description: always having the value 1.
  dtype: int64

#### train-cg
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. This can have the values "gpt2", "real", "scigen" or "galactica".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
  dtype: int64

#### ood_gpt3
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: src
  description: indicator of the source of the paper. Has the value "gpt3".
  dtype: string
- name: label
  description: always having the value 1.
  dtype: int64

#### ood_real
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: src
  description: indicator of the source of the paper. Has the value "ood_real".
  dtype: string
- name: label
  description: always having the value 0.
  dtype: int64

#### test-cc
  - name: id
  description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
  dtype: string
- name: year
  description: year of the publication as given by the arXiv database.
  dtype: string
- name: title
  description: title of the paper given by the arXiv database.
  dtype: string
- name: abstract
  description: abstract of the paper given by the arXiv database.
  dtype: string
- name: introduction
  description: introduction section of the paper. extracted by the PDF parser.
  dtype: string
- name: conclusion
  description: conclusion section of the paper. extracted by the PDF parser.
  dtype: string
- name: categories
  description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
  dtype: string
- name: src
  description: indicator of the source of the paper. Always has the value "chatgpt-paraphrased".
  dtype: string
- name: paraphrased_sections
  description: indicator of which sections are paraphrased. Can have the values "introduction", "conclusion", "introduction, conclusion", "abstract, introduction, conclusion".
  dtype: string
- name: label
  description: 0 for real/human-written papers and 1 for fake/machine-generated papers. Always has the value 1.
  dtype: int64

### Data Splits

Table: Overview of the datasets used to train and evaluate the classifiers.

| Dataset                              | arXiv  | ChatGPT | GPT-2  | SCIgen | Galactica | GPT-3  | ChatGPT (co-created) |
|--------------------------------------|--------|---------|--------|--------|-----------|--------|-----------------------|
| Standard train (TRAIN)               | 8k     | 2k      | 2k     | 2k     | 2k        | -      | -                     |
| Standard train subset (TRAIN-SUB)    | 4k     | 1k      | 1k     | 1k     | 1k        | -      | -                     |
| TRAIN without ChatGPT (TRAIN-CG)     | 8k     | -       | 2k     | 2k     | 2k        | -      | -                     |
| TRAIN plus GPT-3 (TRAIN+GPT3)        | 8k     | 2k      | 2k     | 2k     | 2k        | 1.2k   | -                     |
| Standard test (TEST)                 | 4k     | 1k      | 1k     | 1k     | 1k        | -      | -                     |
| Out-of-domain GPT-3 only (OOD-GPT3)  | -      | -       | -      | -      | -         | 1k     | -                     |
| Out-of-domain real (OOD-REAL)        | 4k (parsing 2) | -  | -      | -      | -         | -      | -                     |
| ChatGPT only (TECG)                  | -      | 1k      | -      | -      | -         | -      | -                     |
| Co-created test (TEST-CC)            | -      | -       | -      | -      | -         | -      | 4k                   |


[More Information Needed]

## Dataset Creation

### Curation Rationale

[More Information Needed]

### Source Data

#### Initial Data Collection and Normalization

[More Information Needed]

#### Who are the source language producers?

[More Information Needed]

### Annotations

#### Annotation process

[More Information Needed]

#### Who are the annotators?

[More Information Needed]

### Personal and Sensitive Information

[More Information Needed]

## Considerations for Using the Data

### Social Impact of Dataset

[More Information Needed]

### Discussion of Biases

[More Information Needed]

### Other Known Limitations

[More Information Needed]

## Additional Information

### Dataset Curators

[More Information Needed]

### Licensing Information

[More Information Needed]

### Citation Information

[More Information Needed]

### Contributions

[More Information Needed]
task_categories:text-classification size_categories:10K<n<100K language:en license:openrail++ scientific paper fake papers science scientific text region:us
{"language": ["en"], "license": "openrail++", "size_categories": ["10K<n<100K"], "task_categories": ["text-classification"], "pretty_name": " A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era (IDMGSP)", "viewer": true, "tags": ["scientific paper", "fake papers", "science", "scientific text"], "dataset_info": [{"config_name": "classifier_input", "features": [{"name": "id", "dtype": "string"}, {"name": "year", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "abstract", "dtype": "string"}, {"name": "introduction", "dtype": "string"}, {"name": "conclusion", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "src", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 70117904, "num_examples": 16000}, {"name": "test", "num_bytes": 34724993, "num_examples": 8000}], "download_size": 32157176, "dataset_size": 104842897}, {"config_name": "tecg", "features": [{"name": "id", "dtype": "string"}, {"name": "year", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "abstract", "dtype": "string"}, {"name": "introduction", "dtype": "string"}, {"name": "conclusion", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "src", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "test", "num_bytes": 2408633, "num_examples": 1000}], "download_size": 582824, "dataset_size": 2408633}, {"config_name": "train+gpt3", "features": [{"name": "id", "dtype": "string"}, {"name": "year", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "abstract", "dtype": "string"}, {"name": "introduction", "dtype": "string"}, {"name": "conclusion", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "src", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 73586425, "num_examples": 17200}], "download_size": 22487536, "dataset_size": 73586425}, {"config_name": "train-cg", "features": [{"name": "id", "dtype": "string"}, {"name": "year", "dtype": "string"}, {"name": "title", "dtype": "string"}, {"name": "abstract", "dtype": "string"}, {"name": "introduction", "dtype": "string"}, {"name": "conclusion", "dtype": "string"}, {"name": "categories", "dtype": "string"}, {"name": "src", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 65261576, "num_examples": 14000}], "download_size": 20272344, "dataset_size": 65261576}, {"config_name": "ood_gpt3", "features": [{"name": "title", "dtype": "string"}, {"name": "abstract", "dtype": "string"}, {"name": "introduction", "dtype": "string"}, {"name": "conclusion", "dtype": "string"}, {"name": "src", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "train", "num_bytes": 3454121, "num_examples": 1200}, {"name": "test", "num_bytes": 2837275, "num_examples": 1000}], "download_size": 1708501, "dataset_size": 6291396}, {"config_name": "ood_real", "features": [{"name": "abstract", "dtype": "string"}, {"name": "introduction", "dtype": "string"}, {"name": "conclusion", "dtype": "string"}, {"name": "src", "dtype": "string"}, {"name": "label", "dtype": "int64"}], "splits": [{"name": "test", "num_bytes": 15808225, "num_examples": 4000}], "download_size": 5336873, "dataset_size": 15808225}]}
TAGS
#task_categories-text-classification #size_categories-10K<n<100K #language-English #license-openrail++ #scientific paper #fake papers #science #scientific text #region-us 

Dataset Card for A Benchmark Dataset for Identifying Machine-Generated Scientific Papers in the LLM Era
=======================================================================================================


Dataset Description
-------------------


* Repository: URL
* Paper: TODO


### Dataset Summary


A benchmark for detecting machine-generated scientific papers based on their abstract, introduction and conclusion sections.


### Supported Tasks and Leaderboards


current benchmark results in terms of accuracy:



### Languages


English


Dataset Structure
-----------------


### Data Instances


Each instance in the dataset corresponds to a row in a CSV file, encompassing the features of a paper, its label, and the paper's source.


### Data Fields


#### classifier\_input


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64


#### train+gpt3


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica", "gpt3".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64


#### tecg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. Always has the value "chatgpt".
dtype: string
* name: label
description: always having the value 1.
dtype: int64


#### train-cg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64


#### ood\_gpt3


* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "gpt3".
dtype: string
* name: label
description: always having the value 1.
dtype: int64


#### ood\_real


dtype: string


* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "ood\_real".
dtype: string
* name: label
description: always having the value 0.
dtype: int64


#### test-cc


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. Always has the value "chatgpt-paraphrased".
dtype: string
* name: paraphrased\_sections
description: indicator of which sections are paraphrased. Can have the values "introduction", "conclusion", "introduction, conclusion", "abstract, introduction, conclusion".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers. Always has the value 1.
dtype: int64


### Data Splits


Table: Overview of the datasets used to train and evaluate the classifiers.



Dataset Creation
----------------


### Curation Rationale


### Source Data


#### Initial Data Collection and Normalization


#### Who are the source language producers?


### Annotations


#### Annotation process


#### Who are the annotators?


### Personal and Sensitive Information


Considerations for Using the Data
---------------------------------


### Social Impact of Dataset


### Discussion of Biases


### Other Known Limitations


Additional Information
----------------------


### Dataset Curators


### Licensing Information


### Contributions



### Dataset Summary


A benchmark for detecting machine-generated scientific papers based on their abstract, introduction and conclusion sections. ### Supported Tasks and Leaderboards


current benchmark results in terms of accuracy: ### Languages


English


Dataset Structure
----------------- ### Data Instances


Each instance in the dataset corresponds to a row in a CSV file, encompassing the features of a paper, its label, and the paper's source. ### Data Fields #### classifier\_input


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64 #### train+gpt3


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica", "gpt3".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64 #### tecg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. Always has the value "chatgpt".
dtype: string
* name: label
description: always having the value 1.
dtype: int64 #### train-cg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64 #### ood\_gpt3


* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "gpt3".
dtype: string
* name: label
description: always having the value 1.
dtype: int64 #### ood\_real


dtype: string


* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "ood\_real".
dtype: string
* name: label
description: always having the value 0.
dtype: int64 #### test-cc


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. Always has the value "chatgpt-paraphrased".
dtype: string
* name: paraphrased\_sections
description: indicator of which sections are paraphrased. Can have the values "introduction", "conclusion", "introduction, conclusion", "abstract, introduction, conclusion".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers. Always has the value 1.
dtype: int64 ### Data Splits


Table: Overview of the datasets used to train and evaluate the classifiers.



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions
TAGS
#task_categories-text-classification #size_categories-10K<n<100K #language-English #license-openrail++ #scientific paper #fake papers #science #scientific text #region-us 
 ### Dataset Summary


A benchmark for detecting machine-generated scientific papers based on their abstract, introduction and conclusion sections. ### Supported Tasks and Leaderboards


current benchmark results in terms of accuracy: ### Languages


English


Dataset Structure
----------------- ### Data Instances


Each instance in the dataset corresponds to a row in a CSV file, encompassing the features of a paper, its label, and the paper's source. ### Data Fields #### classifier\_input


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64 #### train+gpt3


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica", "gpt3".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64 #### tecg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. Always has the value "chatgpt".
dtype: string
* name: label
description: always having the value 1.
dtype: int64 #### train-cg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64 #### ood\_gpt3


* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "gpt3".
dtype: string
* name: label
description: always having the value 1.
dtype: int64 #### ood\_real


dtype: string


* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "ood\_real".
dtype: string
* name: label
description: always having the value 0.
dtype: int64 #### test-cc


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. Always has the value "chatgpt-paraphrased".
dtype: string
* name: paraphrased\_sections
description: indicator of which sections are paraphrased. Can have the values "introduction", "conclusion", "introduction, conclusion", "abstract, introduction, conclusion".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers. Always has the value 1.
dtype: int64 ### Data Splits


Table: Overview of the datasets used to train and evaluate the classifiers.



Dataset Creation
---------------- ### Curation Rationale ### Source Data #### Initial Data Collection and Normalization #### Who are the source language producers? ### Annotations #### Annotation process #### Who are the annotators? ### Personal and Sensitive Information


Considerations for Using the Data
--------------------------------- ### Social Impact of Dataset ### Discussion of Biases ### Other Known Limitations


Additional Information
---------------------- ### Dataset Curators ### Licensing Information ### Contributions
passage: TAGS
#task_categories-text-classification #size_categories-10K<n<100K #language-English #license-openrail++ #scientific paper #fake papers #science #scientific text #region-us 
### Dataset Summary


A benchmark for detecting machine-generated scientific papers based on their abstract, introduction and conclusion sections.### Supported Tasks and Leaderboards


current benchmark results in terms of accuracy:### Languages


English


Dataset Structure
-----------------### Data Instances


Each instance in the dataset corresponds to a row in a CSV file, encompassing the features of a paper, its label, and the paper's source.### Data Fields#### classifier\_input


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64 passage: #### train+gpt3


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "chatgpt", "gpt2", "real", "scigen" or "galactica", "gpt3".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64#### tecg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. Always has the value "chatgpt".
dtype: string
* name: label
description: always having the value 1.
dtype: int64 passage: #### train-cg


* name: id
description: The ID of the provided paper corresponds to the identifier assigned by the arXiv database if the paper's source is marked as "real".
dtype: string
* name: year
description: year of the publication as given by the arXiv database.
dtype: string
* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: categories
description: topics/domains of the paper given by the arXiv database. This field is null if the src field is not "real".
dtype: string
* name: src
description: indicator of the source of the paper. This can have the values "gpt2", "real", "scigen" or "galactica".
dtype: string
* name: label
description: 0 for real/human-written papers and 1 for fake/machine-generated papers.
dtype: int64#### ood\_gpt3


* name: title
description: title of the paper given by the arXiv database.
dtype: string
* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "gpt3".
dtype: string
* name: label
description: always having the value 1.
dtype: int64#### ood\_real


dtype: string


* name: abstract
description: abstract of the paper given by the arXiv database.
dtype: string
* name: introduction
description: introduction section of the paper. extracted by the PDF parser.
dtype: string
* name: conclusion
description: conclusion section of the paper. extracted by the PDF parser.
dtype: string
* name: src
description: indicator of the source of the paper. Has the value "ood\_real".
dtype: string
* name: label
description: always having the value 0.
dtype: int64