# Self-instruct-starcoder

## Table of Contents
- [Summary](#summary)
- [Our approach](#our-approach)
- [Dataset generation](#dataset-generation)
- [Dataset quality](#dataset-quality)
- [Post-processing](#post-processing)
  - [Self-consistency](#self-consistency)
  - [Uniqueness](#uniqueness)
  - [Compile](#compile)
- [Dataset structure](#dataset-structure)
- [Space](#space)

## Summary

Self-instruct-starcoder is a dataset that was generated by prompting starcoder to generate new instructions based on some human-written seed instructions.
The underlying process is explained in the paper [self-instruct](https://arxiv.org/abs/2212.10560). This algorithm gave birth to famous machine generated 
datasets such as [Alpaca](https://github.com/tatsu-lab/stanford_alpaca) and [Code Alpaca](https://github.com/sahil280114/codealpaca) which are two datasets
obtained by prompting OpenAI `text-davinci-003` engine. 

## Our approach
While our method is similar to self-instruct and stanford alpaca, we included some relevant modifications to the pipeline to account for what we wanted. 
- Rather than using `text-davinci-003`, we chose to prompt [StarCoder](https://arxiv.org/abs/2305.06161) which is a 10x smaller LLM developed for code use cases. However, it is possible to use any decoder based LLM on the hub.
- We changed our seed tasks in order to have the model generate code related tasks. We completed the seed tasks from code alpaca with 20 additional algorithm instructions.
- We switched from the generation format `"instruction":` - `"input":` - `"output":` to the format `"instruction":` - `"output":` by concatenating each instruction and its input under the
keyword `instruction`. We did so because the previous prompting format tended to make the model generate test cases as input and their solution as output, which is not what we wanted.
- Finally, we incorporated the possibility to change the trigger word in the prompt. We thus replaced the `"instruction" :` keyword by `"Here is the correct solution to the problem ":` which
resulted into much better generated instructions.

## Dataset generation
The generation of the dataset was time consuming and we chose our parameters to limit the computational burden of our method. 
- Number of examples in context : 4
  - 2 seed instructions
  - 2 machine generated instructions
- Number of instructions to generate : 5000
- Stop words used in the generation : ["\n20", "20.", "20 ."]
- Similarity threshold for rouge score : 0.7

## Dataset quality
StarCoder, while being a great model is not as capable as `text-davinci-003`. In the generation, the model quickly reach sort of a ceiling in terms of creativity.
There are many instructions that are similar to each other, but it should not bother since they are not phrased the same.

## Post-processing
Post-processing is an important part of the pipeline since it improves the quality of the dataset despite the fact that it implies getting rid of some examples. First we
need to identify what we want to avoid :
- A generated solution which does not answer to the corresponding instruction
- An instruction that is too similar to another one.

### Self-consistency
We imagined a process that we named **self-consistency**. The idea is to reverse-prompt the model to see if it can generate a sound instruction that corresponds to the 
solution (output) it is prompted with. This is a particularly difficult few-shot task, and unfortunately StarCoder does not perform incredibly well on it. With a few-shot parameters of `4` 
(all being seed tasks), the model is able to recover 1135 instructions out of 5003, which amount for 22.6% of the raw dataset. Fortunately, the inability for starcoder to generate instructions for some 
solutions does not mean we should get rid of them. For the solutions (outputs) with generated instructions, we can compare these with the ground truth. For that we can use [Sentence-BERT](https://arxiv.org/abs/1908.10084) because the comparison should focus the meaning
rather than the word to word similarity ratio. We have about 771 instructions (~68%) with a similarity score >= 0.5 with their ground truth. These can be seen as high quality examples, they form the `curated` set.

<p align="center">
    <img src="https://huggingface.co/datasets/codeparrot/self-instruct-starcoder/resolve/main/output.png" alt="drawing" width="300", height="300"/>
</p> 

### Uniqueness
Another approach that can be used to clean the raw dataset is to focus on distinct instructions. For a given instruction, we go through all the instructions generated before it to see if there is one with a similarity score >= 0.5.
If it is the case, we remove that instruction. This process removes about 94% of the raw dataset, the remaining instructions form the `unique` set. 

### Compile
We also decided to build a set which contains solely the example featuring a code written in python 3 which does not code a compilation error.

## Dataset structure

```python
from datasets import load_dataset

dataset = load_dataset("codeparrot/self-instruct-starcoder")
DatasetDict({
    compile: Dataset({
        features: ['instruction', 'output', 'most_similar', 'avg_similarity_score'],
        num_rows: 3549
    })
    curated: Dataset({
        features: ['instruction', 'output', 'most_similar', 'avg_similarity_score'],
        num_rows: 771
    })
    raw: Dataset({
        features: ['instruction', 'output', 'most_similar', 'avg_similarity_score'],
        num_rows: 5003
    })
    unique: Dataset({
        features: ['instruction', 'output', 'most_similar', 'avg_similarity_score'],
        num_rows: 308
    })
}))
```
|Field|Type|Description|
|---|---|---|
|instruction|string|Instruction|
|output|string|Answer to the instruction|
|most_similar|string|Dictionnary containing the 10 most similar instructions generated before the current instruction along with the similarity scores|
|avg_similarity_score|float64| Average similarity score|

## Additional resources

- [Space(self-instruct-starcoder)](https://huggingface.co/spaces/codeparrot/self-instruct-starcoder)
- [Github Repository](https://github.com/ArmelRandy/Self-instruct)

## Citation

```
@misc{title={Self-Instruct-StarCoder},
      author={Zebaze, Armel Randy},
      doi={https://doi.org/10.57967/hf/0790},
}
```
task_categories:text2text-generation size_categories:1K<n<10K language:en license:bigscience-openrail-m code arxiv:2212.10560 arxiv:2305.06161 arxiv:1908.10084 doi:10.57967/hf/0790 region:us
{"language": ["en"], "license": "bigscience-openrail-m", "size_categories": ["1K<n<10K"], "task_categories": ["text2text-generation"], "dataset_info": {"features": [{"name": "instruction", "dtype": "string"}, {"name": "output", "dtype": "string"}, {"name": "most_similar", "dtype": "string"}, {"name": "avg_similarity_score", "dtype": "float64"}], "splits": [{"name": "curated", "num_bytes": 1937514, "num_examples": 771}, {"name": "raw", "num_bytes": 12969008, "num_examples": 5003}, {"name": "unique", "num_bytes": 786771, "num_examples": 308}, {"name": "compile", "num_bytes": 9048805, "num_examples": 3549}], "download_size": 10935008, "dataset_size": 24742098}, "tags": ["code"]}
TAGS
#task_categories-text2text-generation #size_categories-1K<n<10K #language-English #license-bigscience-openrail-m #code #arxiv-2212.10560 #arxiv-2305.06161 #arxiv-1908.10084 #doi-10.57967/hf/0790 #region-us 

Self-instruct-starcoder
=======================


Table of Contents
-----------------


* Summary
* Our approach
* Dataset generation
* Dataset quality
* Post-processing
	+ Self-consistency
	+ Uniqueness
	+ Compile
* Dataset structure
* Space


Summary
-------


Self-instruct-starcoder is a dataset that was generated by prompting starcoder to generate new instructions based on some human-written seed instructions.
The underlying process is explained in the paper self-instruct. This algorithm gave birth to famous machine generated
datasets such as Alpaca and Code Alpaca which are two datasets
obtained by prompting OpenAI 'text-davinci-003' engine.


Our approach
------------


While our method is similar to self-instruct and stanford alpaca, we included some relevant modifications to the pipeline to account for what we wanted.


* Rather than using 'text-davinci-003', we chose to prompt StarCoder which is a 10x smaller LLM developed for code use cases. However, it is possible to use any decoder based LLM on the hub.
* We changed our seed tasks in order to have the model generate code related tasks. We completed the seed tasks from code alpaca with 20 additional algorithm instructions.
* We switched from the generation format '"instruction":' - '"input":' - '"output":' to the format '"instruction":' - '"output":' by concatenating each instruction and its input under the
keyword 'instruction'. We did so because the previous prompting format tended to make the model generate test cases as input and their solution as output, which is not what we wanted.
* Finally, we incorporated the possibility to change the trigger word in the prompt. We thus replaced the '"instruction" :' keyword by '"Here is the correct solution to the problem ":' which
resulted into much better generated instructions.


Dataset generation
------------------


The generation of the dataset was time consuming and we chose our parameters to limit the computational burden of our method.


* Number of examples in context : 4
	+ 2 seed instructions
	+ 2 machine generated instructions
* Number of instructions to generate : 5000
* Stop words used in the generation : ["\n20", "20.", "20 ."]
* Similarity threshold for rouge score : 0.7


Dataset quality
---------------


StarCoder, while being a great model is not as capable as 'text-davinci-003'. In the generation, the model quickly reach sort of a ceiling in terms of creativity.
There are many instructions that are similar to each other, but it should not bother since they are not phrased the same.


Post-processing
---------------


Post-processing is an important part of the pipeline since it improves the quality of the dataset despite the fact that it implies getting rid of some examples. First we
need to identify what we want to avoid :


* A generated solution which does not answer to the corresponding instruction
* An instruction that is too similar to another one.


### Self-consistency


We imagined a process that we named self-consistency. The idea is to reverse-prompt the model to see if it can generate a sound instruction that corresponds to the
solution (output) it is prompted with. This is a particularly difficult few-shot task, and unfortunately StarCoder does not perform incredibly well on it. With a few-shot parameters of '4'
(all being seed tasks), the model is able to recover 1135 instructions out of 5003, which amount for 22.6% of the raw dataset. Fortunately, the inability for starcoder to generate instructions for some
solutions does not mean we should get rid of them. For the solutions (outputs) with generated instructions, we can compare these with the ground truth. For that we can use Sentence-BERT because the comparison should focus the meaning
rather than the word to word similarity ratio. We have about 771 instructions (~68%) with a similarity score >= 0.5 with their ground truth. These can be seen as high quality examples, they form the 'curated' set.



![](URL alt=)



### Uniqueness


Another approach that can be used to clean the raw dataset is to focus on distinct instructions. For a given instruction, we go through all the instructions generated before it to see if there is one with a similarity score >= 0.5.
If it is the case, we remove that instruction. This process removes about 94% of the raw dataset, the remaining instructions form the 'unique' set.


### Compile


We also decided to build a set which contains solely the example featuring a code written in python 3 which does not code a compilation error.


Dataset structure
-----------------


Field: instruction, Type: string, Description: Instruction
Field: output, Type: string, Description: Answer to the instruction
Field: most\_similar, Type: string, Description: Dictionnary containing the 10 most similar instructions generated before the current instruction along with the similarity scores
Field: avg\_similarity\_score, Type: float64, Description: Average similarity score


Additional resources
--------------------


* Space(self-instruct-starcoder)
* Github Repository



### Self-consistency


We imagined a process that we named self-consistency. The idea is to reverse-prompt the model to see if it can generate a sound instruction that corresponds to the
solution (output) it is prompted with. This is a particularly difficult few-shot task, and unfortunately StarCoder does not perform incredibly well on it. With a few-shot parameters of '4'
(all being seed tasks), the model is able to recover 1135 instructions out of 5003, which amount for 22.6% of the raw dataset. Fortunately, the inability for starcoder to generate instructions for some
solutions does not mean we should get rid of them. For the solutions (outputs) with generated instructions, we can compare these with the ground truth. For that we can use Sentence-BERT because the comparison should focus the meaning
rather than the word to word similarity ratio. We have about 771 instructions (~68%) with a similarity score >= 0.5 with their ground truth. These can be seen as high quality examples, they form the 'curated' set.



![](URL alt=) ### Uniqueness


Another approach that can be used to clean the raw dataset is to focus on distinct instructions. For a given instruction, we go through all the instructions generated before it to see if there is one with a similarity score >= 0.5.
If it is the case, we remove that instruction. This process removes about 94% of the raw dataset, the remaining instructions form the 'unique' set. ### Compile


We also decided to build a set which contains solely the example featuring a code written in python 3 which does not code a compilation error.


Dataset structure
-----------------


Field: instruction, Type: string, Description: Instruction
Field: output, Type: string, Description: Answer to the instruction
Field: most\_similar, Type: string, Description: Dictionnary containing the 10 most similar instructions generated before the current instruction along with the similarity scores
Field: avg\_similarity\_score, Type: float64, Description: Average similarity score


Additional resources
--------------------


* Space(self-instruct-starcoder)
* Github Repository
TAGS
#task_categories-text2text-generation #size_categories-1K<n<10K #language-English #license-bigscience-openrail-m #code #arxiv-2212.10560 #arxiv-2305.06161 #arxiv-1908.10084 #doi-10.57967/hf/0790 #region-us 
 ### Self-consistency


We imagined a process that we named self-consistency. The idea is to reverse-prompt the model to see if it can generate a sound instruction that corresponds to the
solution (output) it is prompted with. This is a particularly difficult few-shot task, and unfortunately StarCoder does not perform incredibly well on it. With a few-shot parameters of '4'
(all being seed tasks), the model is able to recover 1135 instructions out of 5003, which amount for 22.6% of the raw dataset. Fortunately, the inability for starcoder to generate instructions for some
solutions does not mean we should get rid of them. For the solutions (outputs) with generated instructions, we can compare these with the ground truth. For that we can use Sentence-BERT because the comparison should focus the meaning
rather than the word to word similarity ratio. We have about 771 instructions (~68%) with a similarity score >= 0.5 with their ground truth. These can be seen as high quality examples, they form the 'curated' set.



![](URL alt=) ### Uniqueness


Another approach that can be used to clean the raw dataset is to focus on distinct instructions. For a given instruction, we go through all the instructions generated before it to see if there is one with a similarity score >= 0.5.
If it is the case, we remove that instruction. This process removes about 94% of the raw dataset, the remaining instructions form the 'unique' set. ### Compile


We also decided to build a set which contains solely the example featuring a code written in python 3 which does not code a compilation error.


Dataset structure
-----------------


Field: instruction, Type: string, Description: Instruction
Field: output, Type: string, Description: Answer to the instruction
Field: most\_similar, Type: string, Description: Dictionnary containing the 10 most similar instructions generated before the current instruction along with the similarity scores
Field: avg\_similarity\_score, Type: float64, Description: Average similarity score


Additional resources
--------------------


* Space(self-instruct-starcoder)
* Github Repository
passage: TAGS
#task_categories-text2text-generation #size_categories-1K<n<10K #language-English #license-bigscience-openrail-m #code #arxiv-2212.10560 #arxiv-2305.06161 #arxiv-1908.10084 #doi-10.57967/hf/0790 #region-us 
### Self-consistency


We imagined a process that we named self-consistency. The idea is to reverse-prompt the model to see if it can generate a sound instruction that corresponds to the
solution (output) it is prompted with. This is a particularly difficult few-shot task, and unfortunately StarCoder does not perform incredibly well on it. With a few-shot parameters of '4'
(all being seed tasks), the model is able to recover 1135 instructions out of 5003, which amount for 22.6% of the raw dataset. Fortunately, the inability for starcoder to generate instructions for some
solutions does not mean we should get rid of them. For the solutions (outputs) with generated instructions, we can compare these with the ground truth. For that we can use Sentence-BERT because the comparison should focus the meaning
rather than the word to word similarity ratio. We have about 771 instructions (~68%) with a similarity score >= 0.5 with their ground truth. These can be seen as high quality examples, they form the 'curated' set.



![](URL alt=)### Uniqueness


Another approach that can be used to clean the raw dataset is to focus on distinct instructions. For a given instruction, we go through all the instructions generated before it to see if there is one with a similarity score >= 0.5.
If it is the case, we remove that instruction. This process removes about 94% of the raw dataset, the remaining instructions form the 'unique' set.