# StarEnCoder

##  Table of Contents

1. [Model Summary](##model-summary)
3. [Training](##training)
4. [Use](##use)
5. [Limitations](##limitations)
6. [License](##license)

## Model Summary

This is an encoder-only model (i.e., bi-directionally self-attentive Transformers) trained on [The Stack](https://huggingface.co/datasets/bigcode/the-stack) dataset.

- **Project Website:** [bigcode-project.org](https://www.bigcode-project.org)
- **Point of Contact:** [contact@bigcode-project.org](mailto:contact@bigcode-project.org)
- **Languages:** 80+ Programming languages


We leveraged the :
- Masked Language Modelling (MLM) and Next Sentence Prediction (NSP) objectives from [BERT](https://arxiv.org/abs/1810.04805).
- Predicted masked-out tokens from an input sentence and whether a pair of sentences occur as neighbors in a document.

## Training

We train for 100,000 steps with a global batch size of 4,096 sequences of a maximum length of 1,024 so that approximately 400B~tokens are observed. This takes roughly two days using 64 NVIDIA A100 GPUs. 
Details about the model architecture are reported in the table below.

| Hyperparameter           | Value     |
|--------------------------|-----------|
| Hidden size              | 768       |
| Intermediate size        | 3072      |
| Max. position embeddings | 1024      |
| Num. of attention heads  | 12        |
| Num. of hidden layers    | 12        |
| Attention                | Multi-head|
| Num. of parameters       | ≈125M     |


## Use

This model is trained on 86 programming languages from GitHub code including GitHub issues and Git Commits, and can be efficiently fine-tuned for both code- and text-related tasks.
We fine-tuned on a token classification task to detect PII and have released [StaPII](https://huggingface.co/bigcode/starpii) model.


## Limitations
There are limitations to consider when using StarEncoder. It is an encoder-only model, which limits its flexibility in certain code generation or completion tasks, 
and it was trained on data containing PII, which could pose privacy concerns. Performance may vary across the 80+ supported programming languages,
particularly for less common ones, and the model might struggle with understanding domains outside programming languages. 

## License

The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement [here](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).
{"language": ["code"], "extra_gated_prompt": "## Model License Agreement\nPlease read the BigCode [OpenRAIL-M license](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) agreement before accepting it.\n  ", "extra_gated_fields": {"I accept the above license agreement, and will use the Model complying with the set of use restrictions and sharing requirements": "checkbox"}}
transformers pytorch bert pretraining code arxiv:1810.04805 endpoints_compatible has_space region:us
TAGS
#transformers #pytorch #bert #pretraining #code #arxiv-1810.04805 #endpoints_compatible #has_space #region-us 

StarEnCoder
===========


Table of Contents
-----------------


1. Model Summary
2. Training
3. Use
4. Limitations
5. License


Model Summary
-------------


This is an encoder-only model (i.e., bi-directionally self-attentive Transformers) trained on The Stack dataset.


* Project Website: URL
* Point of Contact: contact@URL
* Languages: 80+ Programming languages


We leveraged the :


* Masked Language Modelling (MLM) and Next Sentence Prediction (NSP) objectives from BERT.
* Predicted masked-out tokens from an input sentence and whether a pair of sentences occur as neighbors in a document.


Training
--------


We train for 100,000 steps with a global batch size of 4,096 sequences of a maximum length of 1,024 so that approximately 400B~tokens are observed. This takes roughly two days using 64 NVIDIA A100 GPUs.
Details about the model architecture are reported in the table below.



Use
---


This model is trained on 86 programming languages from GitHub code including GitHub issues and Git Commits, and can be efficiently fine-tuned for both code- and text-related tasks.
We fine-tuned on a token classification task to detect PII and have released StaPII model.


Limitations
-----------


There are limitations to consider when using StarEncoder. It is an encoder-only model, which limits its flexibility in certain code generation or completion tasks,
and it was trained on data containing PII, which could pose privacy concerns. Performance may vary across the 80+ supported programming languages,
particularly for less common ones, and the model might struggle with understanding domains outside programming languages.


License
-------


The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement here.



TAGS
#transformers #pytorch #bert #pretraining #code #arxiv-1810.04805 #endpoints_compatible #has_space #region-us 

TAGS
#transformers #pytorch #bert #pretraining #code #arxiv-1810.04805 #endpoints_compatible #has_space #region-us 
