

# pythia-160m-hq-emails-v4

This model is a fine-tuned version of [EleutherAI/pythia-160m-deduped](https://huggingface.co/EleutherAI/pythia-160m-deduped) on the postbot/multi-emails-hq dataset.
It achieves the following results on the evaluation set:
- Loss: 2.2856
- Accuracy: 0.6113
- perplexity: 9.8313

## Model description

this is v4

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0006
- train_batch_size: 4
- eval_batch_size: 1
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 32
- total_train_batch_size: 128
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 4.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:--------:|
| 2.412         | 0.99  | 76   | 2.5027          | 0.5458   |
| 1.9702        | 1.99  | 152  | 2.2757          | 0.5850   |
| 1.4628        | 2.99  | 228  | 2.2162          | 0.6082   |
| 1.1662        | 3.99  | 304  | 2.2856          | 0.6113   |


### Framework versions

- Transformers 4.27.0.dev0
- Pytorch 1.13.1+cu117
- Datasets 2.8.0
- Tokenizers 0.13.1
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_postbot__pythia-160m-hq-emails)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 25.12   |
| ARC (25-shot)         | 23.12          |
| HellaSwag (10-shot)   | 30.05    |
| MMLU (5-shot)         | 26.58         |
| TruthfulQA (0-shot)   | 45.51   |
| Winogrande (5-shot)   | 50.28   |
| GSM8K (5-shot)        | 0.0        |
| DROP (3-shot)         | 0.31         |

{"language": ["en"], "license": "apache-2.0", "tags": ["generated_from_trainer"], "datasets": ["postbot/multi-emails-hq"], "metrics": ["accuracy"], "widget": [{"text": "Good Morning Professor Beans,\nHope you are doing well. I just wanted to reach out and ask if differential calculus will be on the exam", "example_title": "email to prof"}, {"text": "Hey <NAME>,\n\nThank you for signing up for my weekly newsletter. Before we get started, you'll have to confirm your email address.", "example_title": "newsletter"}, {"text": "Hi <NAME>,\n\nI hope this email finds you well. I wanted to reach out and ask about office hours", "example_title": "office hours"}, {"text": "Greetings <NAME>,\n\nI hope you had a splendid evening at the Company sausage eating festival. I am reaching out because", "example_title": "festival"}, {"text": "Good Morning Harold,\n\nI was wondering when the next", "example_title": "event"}, {"text": "URGENT - I need the TPS reports", "example_title": "URGENT"}, {"text": "Hi Archibald,\n\nI hope this email finds you extremely well.", "example_title": "emails that find you"}, {"text": "Hello there.\n\nI just wanted to reach out and check in to", "example_title": "checking in"}, {"text": "Hello <NAME>,\n\nI hope this email finds you well. I wanted to reach out and see if you've enjoyed your time with us", "example_title": "work well"}, {"text": "Hi <NAME>,\n\nI hope this email finds you well. I wanted to reach out and see if we could catch up", "example_title": "catch up"}, {"text": "I'm <NAME> and I just moved into the area and wanted to reach out and get some details on where I could get groceries and", "example_title": "grocery"}], "inference": {"parameters": {"min_length": 16, "max_length": 64, "no_repeat_ngram_size": 4, "do_sample": true, "top_k": 40, "top_p": 0.95, "repetition_penalty": 3.5}}, "pipeline_tag": "text-generation", "base_model": "EleutherAI/pythia-160m-deduped", "model-index": [{"name": "pythia-160m-hq-emails-v4", "results": [{"task": {"type": "text-generation", "name": "Causal Language Modeling"}, "dataset": {"name": "postbot/multi-emails-hq", "type": "postbot/multi-emails-hq"}, "metrics": [{"type": "accuracy", "value": 0.611281497151223, "name": "Accuracy"}]}]}]}
transformers pytorch safetensors gpt_neox text-generation generated_from_trainer en dataset:postbot/multi-emails-hq base_model:EleutherAI/pythia-160m-deduped license:apache-2.0 model-index autotrain_compatible endpoints_compatible has_space text-generation-inference region:us
TAGS
#transformers #pytorch #safetensors #gpt_neox #text-generation #generated_from_trainer #en #dataset-postbot/multi-emails-hq #base_model-EleutherAI/pythia-160m-deduped #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 

pythia-160m-hq-emails-v4
========================


This model is a fine-tuned version of EleutherAI/pythia-160m-deduped on the postbot/multi-emails-hq dataset.
It achieves the following results on the evaluation set:


* Loss: 2.2856
* Accuracy: 0.6113
* perplexity: 9.8313


Model description
-----------------


this is v4


Intended uses & limitations
---------------------------


More information needed


Training and evaluation data
----------------------------


More information needed


Training procedure
------------------


### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0006
* train\_batch\_size: 4
* eval\_batch\_size: 1
* seed: 42
* distributed\_type: multi-GPU
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 128
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_ratio: 0.05
* num\_epochs: 4.0
* mixed\_precision\_training: Native AMP


### Training results



### Framework versions


* Transformers 4.27.0.dev0
* Pytorch 1.13.1+cu117
* Datasets 2.8.0
* Tokenizers 0.13.1


Open LLM Leaderboard Evaluation Results
=======================================


Detailed results can be found here




### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0006
* train\_batch\_size: 4
* eval\_batch\_size: 1
* seed: 42
* distributed\_type: multi-GPU
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 128
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_ratio: 0.05
* num\_epochs: 4.0
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.27.0.dev0
* Pytorch 1.13.1+cu117
* Datasets 2.8.0
* Tokenizers 0.13.1


Open LLM Leaderboard Evaluation Results
=======================================


Detailed results can be found here
TAGS
#transformers #pytorch #safetensors #gpt_neox #text-generation #generated_from_trainer #en #dataset-postbot/multi-emails-hq #base_model-EleutherAI/pythia-160m-deduped #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0006
* train\_batch\_size: 4
* eval\_batch\_size: 1
* seed: 42
* distributed\_type: multi-GPU
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 128
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_ratio: 0.05
* num\_epochs: 4.0
* mixed\_precision\_training: Native AMP ### Training results ### Framework versions


* Transformers 4.27.0.dev0
* Pytorch 1.13.1+cu117
* Datasets 2.8.0
* Tokenizers 0.13.1


Open LLM Leaderboard Evaluation Results
=======================================


Detailed results can be found here
TAGS
#transformers #pytorch #safetensors #gpt_neox #text-generation #generated_from_trainer #en #dataset-postbot/multi-emails-hq #base_model-EleutherAI/pythia-160m-deduped #license-apache-2.0 #model-index #autotrain_compatible #endpoints_compatible #has_space #text-generation-inference #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* learning\_rate: 0.0006
* train\_batch\_size: 4
* eval\_batch\_size: 1
* seed: 42
* distributed\_type: multi-GPU
* gradient\_accumulation\_steps: 32
* total\_train\_batch\_size: 128
* optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
* lr\_scheduler\_type: cosine
* lr\_scheduler\_warmup\_ratio: 0.05
* num\_epochs: 4.0
* mixed\_precision\_training: Native AMP### Training results### Framework versions


* Transformers 4.27.0.dev0
* Pytorch 1.13.1+cu117
* Datasets 2.8.0
* Tokenizers 0.13.1


Open LLM Leaderboard Evaluation Results
=======================================


Detailed results can be found here