
# Machine-essays generation pipeline

Please check out our [github repo](https://github.com/huhailinguist/ArguGPT).

This document only introduces how we collected **machine-generated essays**. 

| model            | timestamp   | # total | # valid | # short | # repetitive | # overlapped |
|------------------|-------------|---------|---------|---------|--------------|--------------|
| gpt2-xl          | Nov, 2019   | 4,573   | 563     | 1,637   | 0            | 2,373        |
| text-babbage-001 | April, 2022 | 917     | 479     | 181     | 240          | 17           |
| text-curie-001   | April, 2022 | 654     | 498     | 15      | 110          | 31           |
| text-davinci-001 | April, 2022 | 632     | 493     | 1       | 41           | 97           |
| text-davinci-002 | April, 2022 | 621     | 495     | 1       | 56           | 69           |
| text-davinci-003 | Nov, 2022   | 1,130   | 1,090   | 0       | 30           | 10           |
| gpt-3.5-turbo    | Mar, 2023   | 1,122   | 1,090   | 0       | 4            | 28           |
| total            | -           | 9,647   | 4,708   | 1,835   | 481          | 2,625        |


## Models

We chose 7 models from GPT family: 1) `gpt2-xl`, 2) `text-babbage-001`, 3) `text-curie-001`, 4) `text-davinci-001`, 5) `text-davinci-002`, 
6) `text-davinci-003`, and 7) `gpt-3.5-turbo`. 
More information about these models can be seen in [OpenAI documentation](https://platform.openai.com/docs/model-index-for-researchers). 

For WECCL and TOEFL, we used all 7 models to generate argumentative essays. 
As for GRE, of which the writing task is more difficult than WECCL and TOEFL, we only used `text-davinci-003` and `gpt-3.5-turbo`. 

**Notes**: Since `gpt2-xl` cannot respond to prompts as InstructGPTs and other later models, 
we fed `gpt2-xl` the prompt along with one beginning sentence randomly extracted from human essays for continuous writing. 
Therefore, the first sentence of each essay generated by `gpt2-xl` is actually human-authored. 

## Prompts selection

Our writing topics are collected from human-WECCL, human-TOEFL, and human-GRE. 
In a writing task, a topic statement is presented for students (or machines) to attack or defend. 
The topic statement here is refered to `ESSAY_PROMPT`, and our added instructions for machine is refered to `ADDED_PROMPT`.

Therefore, our prompt format is as follow: `ESSAY_PROMPT` + `ADDED_PROMPT`. 

For instance, 
- `ESSAY_PROMPT`: It is better to have broad knowledge of many academic subjects than to specialize in one specific subject. 
- `ADDED_PROMPT`: Do you agree or disagree? Use specific reasons and examples to support your answer. Write an essay of roughly {300/400/500} words.

We asked the machine to write 300 words for writing tasks in WECCL, 400 for TOEFL, and 500 for GRE. 

## Essays filtering, preprocessing, and automated scoring

We then filtered out the essays that are short, repetitive and overlapped. 
- Short: we set the threshold of 50 words for `gpt2-xl`, and 100 words for others.
- Repetitive: 40% of sentences are *similar*.
- Overlapped: 40% of sentences are *similar* with any other essay already generated.
- Definition of *similar*: "I like a dog." and "I don't like a cat." have 3 words in common. The similarity therefore is 6 / 9 = 0.67. If the similarity is greater than 0.8, the two sentences are *similar*.

We deleted "As an AI model, ..." generated by gpt-3.5-turbo. 
And we used [YouDao automated scoring system](https://ai.youdao.com/) to score all the essays, 
and categorized them into low, mid, and high levels.

## Citation

Please cite our work [arXiv:2304.07666](https://arxiv.org/abs/2304.07666) as  

```
@misc{liu2023argugpt,
      title={ArguGPT: evaluating, understanding and identifying argumentative essays generated by GPT models}, 
      author={Yikang Liu and Ziyin Zhang and Wanyang Zhang and Shisen Yue and Xiaojing Zhao and Xinyuan Cheng and Yiwen Zhang and Hai Hu},
      year={2023},
      eprint={2304.07666},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

task_categories:text-classification size_categories:1K<n<10K language:en license:cc AIGC for education arxiv:2304.07666 region:us
{"language": ["en"], "license": "cc", "size_categories": ["1K<n<10K"], "task_categories": ["text-classification"], "tags": ["AIGC for education"]}
TAGS
#task_categories-text-classification #size_categories-1K<n<10K #language-English #license-cc #AIGC for education #arxiv-2304.07666 #region-us 

Machine-essays generation pipeline
==================================


Please check out our github repo.


This document only introduces how we collected machine-generated essays.



Models
------


We chose 7 models from GPT family: 1) 'gpt2-xl', 2) 'text-babbage-001', 3) 'text-curie-001', 4) 'text-davinci-001', 5) 'text-davinci-002',
6) 'text-davinci-003', and 7) 'gpt-3.5-turbo'.
More information about these models can be seen in OpenAI documentation.


For WECCL and TOEFL, we used all 7 models to generate argumentative essays.
As for GRE, of which the writing task is more difficult than WECCL and TOEFL, we only used 'text-davinci-003' and 'gpt-3.5-turbo'.


Notes: Since 'gpt2-xl' cannot respond to prompts as InstructGPTs and other later models,
we fed 'gpt2-xl' the prompt along with one beginning sentence randomly extracted from human essays for continuous writing.
Therefore, the first sentence of each essay generated by 'gpt2-xl' is actually human-authored.


Prompts selection
-----------------


Our writing topics are collected from human-WECCL, human-TOEFL, and human-GRE.
In a writing task, a topic statement is presented for students (or machines) to attack or defend.
The topic statement here is refered to 'ESSAY\_PROMPT', and our added instructions for machine is refered to 'ADDED\_PROMPT'.


Therefore, our prompt format is as follow: 'ESSAY\_PROMPT' + 'ADDED\_PROMPT'.


For instance,


* 'ESSAY\_PROMPT': It is better to have broad knowledge of many academic subjects than to specialize in one specific subject.
* 'ADDED\_PROMPT': Do you agree or disagree? Use specific reasons and examples to support your answer. Write an essay of roughly {300/400/500} words.


We asked the machine to write 300 words for writing tasks in WECCL, 400 for TOEFL, and 500 for GRE.


Essays filtering, preprocessing, and automated scoring
------------------------------------------------------


We then filtered out the essays that are short, repetitive and overlapped.


* Short: we set the threshold of 50 words for 'gpt2-xl', and 100 words for others.
* Repetitive: 40% of sentences are *similar*.
* Overlapped: 40% of sentences are *similar* with any other essay already generated.
* Definition of *similar*: "I like a dog." and "I don't like a cat." have 3 words in common. The similarity therefore is 6 / 9 = 0.67. If the similarity is greater than 0.8, the two sentences are *similar*.


We deleted "As an AI model, ..." generated by gpt-3.5-turbo.
And we used YouDao automated scoring system to score all the essays,
and categorized them into low, mid, and high levels.


Please cite our work arXiv:2304.07666 as



TAGS
#task_categories-text-classification #size_categories-1K<n<10K #language-English #license-cc #AIGC for education #arxiv-2304.07666 #region-us 

passage: TAGS
#task_categories-text-classification #size_categories-1K<n<10K #language-English #license-cc #AIGC for education #arxiv-2304.07666 #region-us 
