
# 🇪🇺 ✍🏻 EuCorrector

The EuCorrector model aims to enhance the drafting quality of documents, aligning with the standards of European publications. It has undergone training in English, focusing on 12 million sentences sourced from documents produced by the European Publication Office over the past 30 years.

## Technical Description

EuCorrector is an advanced model, built upon the T5 small Transformer framework.
It's a sequence-to-sequence model, fine-tuned using a synthetic dataset.
This dataset is inspired by the [back-translation](https://aclanthology.org/P16-1009/)) technique, similar to that used in Google's research for generating synthetic data for grammatical error correction  [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://aclanthology.org/2021.bea-1.4/).
The error model within EuCorrector was developed using a manually corrected dataset comprising 70,000 sentences.

## Hyperparameters & Specifications

- Synthetic Dataset: 12 million sentences
- Maximum Token Length: 64
- Optimization Method: AdamW
  - Beta Values: (0.9, 0.999)
  - Epsilon Value: 1e-08
- Learning Scheduler: Cosine
- Gradient Accumulation Steps: 32
- Learning Rate: 1e-04
- Training Epochs: 1

- Training Resources:
  - Utilized EuroHPC Meluxina node with 4 NVIDIA A100 GPUs

## Current Evaluation Status

The model is being evaluated using the BLEU, ROUGE, and Levenshtein metrics.

### Future Plans & Improvements

I plans to reduce the learning rate, drawing on approaches used in models like grammarly/coedit-large for enhanced performance.

## 📚 References:
- Sennrich, R., Haddow, B., & Birch, A. (2016). [Improving Neural Machine Translation Models with Monolingual Data](https://aclanthology.org/P16-1009/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL).
- Bryant, C., & Briscoe, T. (2021). [Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models](https://aclanthology.org/2021.bea-1.4/). In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications.

## Author:
- Sébastien Campion
{"language": ["en"], "license": "eupl-1.1", "tags": ["grammar"], "widget": [{"text": "Invites the European Commission to provide support in the elaboration of such EU position;", "example_title": "Example 1"}, {"text": "Notes that compliance with EU legislation on data protection should be the precondition to this digital transformation of health and care in the Digital Single Market", "example_title": "Example 2"}, {"text": "to wind up the debate on the statement by the Vice-President of the Commission / High Representative of the Union for Foreign Affairs and Security Policy the Council and the Commission.", "example_title": "Example 3"}]}
transformers safetensors t5 text2text-generation grammar en license:eupl-1.1 autotrain_compatible endpoints_compatible has_space text-generation-inference region:us