

# emailgen-pythia-410m-deduped

[![colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/pszemraj/94b0e6b95437896f800a65ae2e5f9ab4/emailgen-pythia-410m-deduped.ipynb
)


This model is a fine-tuned version of [EleutherAI/pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped) on email data.
It achieves the following results on the evaluation set:
- Loss: 2.1018
- Accuracy: 0.6157
- perplexity: 8.181

## Model description

- fine-tuned on dataset of emails for 4 epochs
- intended use: "text completion" of partially written emails


## Usage example



```python
from transformers import pipeline

model_tag = "postbot/emailgen-pythia-410m-deduped"
generator = pipeline(
    "text-generation",
    model=model_tag,
)

prompt = """
Hello, 

Following up on the bubblegum shipment."""

result = generator(
    prompt,
)  # generate
print(result[0]["generated_text"])
```

---



# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_postbot__emailgen-pythia-410m-deduped)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 26.65   |
| ARC (25-shot)         | 27.9          |
| HellaSwag (10-shot)   | 40.04    |
| MMLU (5-shot)         | 27.35         |
| TruthfulQA (0-shot)   | 38.2   |
| Winogrande (5-shot)   | 52.09   |
| GSM8K (5-shot)        | 0.0        |
| DROP (3-shot)         | 0.99         |

{"language": ["en"], "license": "apache-2.0", "tags": ["generated_from_trainer"], "datasets": ["postbot/multi-emails-hq"], "metrics": ["accuracy"], "widget": [{"text": "Good Morning Professor Beans,\nHope you are doing well. I just wanted to reach out and ask if differential calculus will be on the exam", "example_title": "email to prof"}, {"text": "Hey <NAME>,\n\nThank you for signing up for my weekly newsletter. Before we get started, you'll have to confirm your email address.", "example_title": "newsletter"}, {"text": "Hi <NAME>,\n\nI hope this email finds you well. I wanted to reach out and ask about office hours", "example_title": "office hours"}, {"text": "Greetings <NAME>,\n\nI hope you had a splendid evening at the Company sausage eating festival. I am reaching out because", "example_title": "festival"}, {"text": "Good Morning Harold,\n\nI was wondering when the next", "example_title": "event"}, {"text": "URGENT - I need the TPS reports", "example_title": "URGENT"}, {"text": "Hi Archibald,\n\nI hope this email finds you extremely well.", "example_title": "emails that find you"}, {"text": "Hello there.\n\nI just wanted to reach out and check in to", "example_title": "checking in"}, {"text": "Hello <NAME>,\n\nI hope this email finds you well. I wanted to reach out and see if you've enjoyed your time with us", "example_title": "work well"}, {"text": "Hi <NAME>,\n\nI hope this email finds you well. I wanted to reach out and see if we could catch up", "example_title": "catch up"}, {"text": "I'm <NAME> and I just moved into the area and wanted to reach out and get some details on where I could get groceries and", "example_title": "grocery"}], "pipeline_tag": "text-generation", "base_model": "EleutherAI/pythia-410m-deduped", "model-index": [{"name": "multi-emails-hq-pythia-410m-deduped-r1", "results": []}]}
transformers pytorch onnx safetensors gpt_neox text-generation generated_from_trainer en dataset:postbot/multi-emails-hq base_model:EleutherAI/pythia-410m-deduped license:apache-2.0 autotrain_compatible endpoints_compatible text-generation-inference region:us
TAGS
#transformers #pytorch #onnx #safetensors #gpt_neox #text-generation #generated_from_trainer #en #dataset-postbot/multi-emails-hq #base_model-EleutherAI/pythia-410m-deduped #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

emailgen-pythia-410m-deduped
============================


![colab](URL)


This model is a fine-tuned version of EleutherAI/pythia-410m-deduped on email data.
It achieves the following results on the evaluation set:


* Loss: 2.1018
* Accuracy: 0.6157
* perplexity: 8.181


Model description
-----------------


* fine-tuned on dataset of emails for 4 epochs
* intended use: "text completion" of partially written emails


Usage example
-------------




---


Open LLM Leaderboard Evaluation Results
=======================================


Detailed results can be found here




TAGS
#transformers #pytorch #onnx #safetensors #gpt_neox #text-generation #generated_from_trainer #en #dataset-postbot/multi-emails-hq #base_model-EleutherAI/pythia-410m-deduped #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 

TAGS
#transformers #pytorch #onnx #safetensors #gpt_neox #text-generation #generated_from_trainer #en #dataset-postbot/multi-emails-hq #base_model-EleutherAI/pythia-410m-deduped #license-apache-2.0 #autotrain_compatible #endpoints_compatible #text-generation-inference #region-us 
