
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# wikitext-103-raw-v1-para-permute-9_default_gpt2_from_scratch

This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on the tyzhu/wikitext-103-raw-v1-para-permute-9 default dataset.
It achieves the following results on the evaluation set:
- Loss: 2.8899
- Accuracy: 0.4557

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10.0

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Accuracy |
|:-------------:|:-----:|:------:|:---------------:|:--------:|
| No log        | 0.05  | 893    | 5.2934          | 0.2371   |
| 5.5908        | 0.1   | 1786   | 4.6313          | 0.2862   |
| 5.5908        | 0.15  | 2679   | 4.2234          | 0.3214   |
| 4.3959        | 0.2   | 3572   | 3.9606          | 0.3463   |
| 4.3959        | 0.25  | 4465   | 3.7982          | 0.3608   |
| 3.9428        | 0.3   | 5358   | 3.6778          | 0.3701   |
| 3.9428        | 0.35  | 6251   | 3.5910          | 0.3777   |
| 3.7139        | 0.4   | 7144   | 3.5189          | 0.3840   |
| 3.7139        | 0.45  | 8037   | 3.4623          | 0.3894   |
| 3.5696        | 0.5   | 8930   | 3.4123          | 0.3937   |
| 3.5696        | 0.55  | 9823   | 3.3721          | 0.3976   |
| 3.4666        | 0.6   | 10716  | 3.3382          | 0.4009   |
| 3.4666        | 0.65  | 11609  | 3.3084          | 0.4032   |
| 3.3893        | 0.7   | 12502  | 3.2817          | 0.4061   |
| 3.3893        | 0.75  | 13395  | 3.2524          | 0.4097   |
| 3.3251        | 0.8   | 14288  | 3.2327          | 0.4112   |
| 3.3251        | 0.85  | 15181  | 3.2120          | 0.4133   |
| 3.2725        | 0.9   | 16074  | 3.1921          | 0.4151   |
| 3.2725        | 0.95  | 16967  | 3.1762          | 0.4170   |
| 3.2265        | 1.0   | 17860  | 3.1644          | 0.4179   |
| 3.2265        | 1.05  | 18753  | 3.1544          | 0.4192   |
| 3.1775        | 1.1   | 19646  | 3.1372          | 0.4211   |
| 3.1775        | 1.15  | 20539  | 3.1283          | 0.4223   |
| 3.1422        | 1.2   | 21432  | 3.1154          | 0.4226   |
| 3.1422        | 1.25  | 22325  | 3.1052          | 0.4245   |
| 3.1155        | 1.3   | 23218  | 3.0972          | 0.4252   |
| 3.1155        | 1.35  | 24111  | 3.0857          | 0.4273   |
| 3.0887        | 1.4   | 25004  | 3.0796          | 0.4272   |
| 3.0887        | 1.45  | 25897  | 3.0700          | 0.4285   |
| 3.0642        | 1.5   | 26790  | 3.0604          | 0.4296   |
| 3.0642        | 1.55  | 27683  | 3.0537          | 0.4306   |
| 3.0426        | 1.6   | 28576  | 3.0472          | 0.4301   |
| 3.0426        | 1.65  | 29469  | 3.0440          | 0.4308   |
| 3.023         | 1.7   | 30362  | 3.0362          | 0.4321   |
| 3.023         | 1.75  | 31255  | 3.0295          | 0.4333   |
| 3.0018        | 1.8   | 32148  | 3.0269          | 0.4330   |
| 3.0018        | 1.85  | 33041  | 3.0233          | 0.4335   |
| 2.9846        | 1.9   | 33934  | 3.0169          | 0.4347   |
| 2.9846        | 1.95  | 34827  | 3.0119          | 0.4356   |
| 2.969         | 2.0   | 35720  | 3.0044          | 0.4365   |
| 2.969         | 2.05  | 36613  | 3.0018          | 0.4365   |
| 2.9357        | 2.1   | 37506  | 3.0004          | 0.4370   |
| 2.9357        | 2.15  | 38399  | 2.9936          | 0.4371   |
| 2.9272        | 2.2   | 39292  | 2.9917          | 0.4386   |
| 2.9272        | 2.25  | 40185  | 2.9864          | 0.4383   |
| 2.9168        | 2.3   | 41078  | 2.9862          | 0.4385   |
| 2.9168        | 2.35  | 41971  | 2.9842          | 0.4390   |
| 2.9036        | 2.4   | 42864  | 2.9796          | 0.4401   |
| 2.9036        | 2.45  | 43757  | 2.9757          | 0.4396   |
| 2.8888        | 2.5   | 44650  | 2.9725          | 0.4401   |
| 2.8888        | 2.55  | 45543  | 2.9698          | 0.4407   |
| 2.8789        | 2.6   | 46436  | 2.9688          | 0.4409   |
| 2.8789        | 2.65  | 47329  | 2.9643          | 0.4414   |
| 2.8672        | 2.7   | 48222  | 2.9620          | 0.4418   |
| 2.8672        | 2.75  | 49115  | 2.9602          | 0.4424   |
| 2.8605        | 2.8   | 50008  | 2.9568          | 0.4427   |
| 2.8605        | 2.85  | 50901  | 2.9549          | 0.4428   |
| 2.8508        | 2.9   | 51794  | 2.9520          | 0.4427   |
| 2.8508        | 2.95  | 52687  | 2.9524          | 0.4429   |
| 2.8423        | 3.0   | 53580  | 2.9519          | 0.4430   |
| 2.8423        | 3.05  | 54473  | 2.9486          | 0.4433   |
| 2.8178        | 3.1   | 55366  | 2.9480          | 0.4437   |
| 2.8178        | 3.15  | 56259  | 2.9469          | 0.4442   |
| 2.8095        | 3.2   | 57152  | 2.9438          | 0.4443   |
| 2.8095        | 3.25  | 58045  | 2.9442          | 0.4452   |
| 2.8045        | 3.3   | 58938  | 2.9399          | 0.4455   |
| 2.8045        | 3.35  | 59831  | 2.9392          | 0.4450   |
| 2.7979        | 3.4   | 60724  | 2.9388          | 0.4456   |
| 2.7979        | 3.45  | 61617  | 2.9384          | 0.4459   |
| 2.7915        | 3.5   | 62510  | 2.9366          | 0.4463   |
| 2.7915        | 3.55  | 63403  | 2.9362          | 0.4457   |
| 2.7841        | 3.6   | 64296  | 2.9330          | 0.4460   |
| 2.7841        | 3.65  | 65189  | 2.9311          | 0.4461   |
| 2.7796        | 3.7   | 66082  | 2.9282          | 0.4467   |
| 2.7796        | 3.75  | 66975  | 2.9287          | 0.4476   |
| 2.7721        | 3.8   | 67868  | 2.9286          | 0.4470   |
| 2.7721        | 3.85  | 68761  | 2.9271          | 0.4466   |
| 2.7675        | 3.9   | 69654  | 2.9271          | 0.4472   |
| 2.7675        | 3.95  | 70547  | 2.9247          | 0.4470   |
| 2.7602        | 4.0   | 71440  | 2.9231          | 0.4478   |
| 2.7602        | 4.05  | 72333  | 2.9255          | 0.4472   |
| 2.7395        | 4.1   | 73226  | 2.9230          | 0.4479   |
| 2.7395        | 4.15  | 74119  | 2.9225          | 0.4480   |
| 2.7347        | 4.2   | 75012  | 2.9211          | 0.4477   |
| 2.7347        | 4.25  | 75905  | 2.9225          | 0.4477   |
| 2.733         | 4.3   | 76798  | 2.9212          | 0.4480   |
| 2.733         | 4.35  | 77691  | 2.9197          | 0.4483   |
| 2.73          | 4.4   | 78584  | 2.9189          | 0.4483   |
| 2.73          | 4.45  | 79477  | 2.9197          | 0.4490   |
| 2.7245        | 4.5   | 80370  | 2.9173          | 0.4493   |
| 2.7245        | 4.55  | 81263  | 2.9170          | 0.4492   |
| 2.7213        | 4.6   | 82156  | 2.9169          | 0.4492   |
| 2.7213        | 4.65  | 83049  | 2.9159          | 0.4494   |
| 2.7154        | 4.7   | 83942  | 2.9139          | 0.4497   |
| 2.7154        | 4.75  | 84835  | 2.9150          | 0.4495   |
| 2.7135        | 4.8   | 85728  | 2.9141          | 0.4497   |
| 2.7135        | 4.85  | 86621  | 2.9131          | 0.4505   |
| 2.7064        | 4.9   | 87514  | 2.9138          | 0.4501   |
| 2.7064        | 4.95  | 88407  | 2.9101          | 0.4505   |
| 2.7023        | 5.0   | 89300  | 2.9123          | 0.4507   |
| 2.7023        | 5.05  | 90193  | 2.9119          | 0.4498   |
| 2.6834        | 5.1   | 91086  | 2.9130          | 0.4501   |
| 2.6834        | 5.15  | 91979  | 2.9128          | 0.4501   |
| 2.6822        | 5.2   | 92872  | 2.9122          | 0.4508   |
| 2.6822        | 5.25  | 93765  | 2.9096          | 0.4508   |
| 2.6791        | 5.3   | 94658  | 2.9113          | 0.4510   |
| 2.6791        | 5.35  | 95551  | 2.9105          | 0.4501   |
| 2.6825        | 5.4   | 96444  | 2.9092          | 0.4509   |
| 2.6825        | 5.45  | 97337  | 2.9075          | 0.4511   |
| 2.6753        | 5.5   | 98230  | 2.9054          | 0.4520   |
| 2.6753        | 5.55  | 99123  | 2.9053          | 0.4513   |
| 2.6738        | 5.6   | 100016 | 2.9043          | 0.4519   |
| 2.6738        | 5.65  | 100909 | 2.9049          | 0.4514   |
| 2.6708        | 5.7   | 101802 | 2.9030          | 0.4517   |
| 2.6708        | 5.75  | 102695 | 2.9062          | 0.4513   |
| 2.6683        | 5.8   | 103588 | 2.9041          | 0.4513   |
| 2.6683        | 5.85  | 104481 | 2.9040          | 0.4518   |
| 2.6641        | 5.9   | 105374 | 2.9039          | 0.4521   |
| 2.6641        | 5.95  | 106267 | 2.9028          | 0.4521   |
| 2.6589        | 6.0   | 107160 | 2.9024          | 0.4521   |
| 2.6589        | 6.05  | 108053 | 2.9060          | 0.4517   |
| 2.6436        | 6.1   | 108946 | 2.9045          | 0.4521   |
| 2.6436        | 6.15  | 109839 | 2.9017          | 0.4523   |
| 2.6426        | 6.2   | 110732 | 2.9026          | 0.4525   |
| 2.6426        | 6.25  | 111625 | 2.9034          | 0.4522   |
| 2.6417        | 6.3   | 112518 | 2.9006          | 0.4526   |
| 2.6417        | 6.35  | 113411 | 2.9027          | 0.4524   |
| 2.6412        | 6.4   | 114304 | 2.9027          | 0.4526   |
| 2.6412        | 6.45  | 115197 | 2.9023          | 0.4528   |
| 2.6391        | 6.5   | 116090 | 2.9012          | 0.4526   |
| 2.6391        | 6.55  | 116983 | 2.9011          | 0.4528   |
| 2.6357        | 6.6   | 117876 | 2.8991          | 0.4532   |
| 2.6357        | 6.65  | 118769 | 2.8992          | 0.4533   |
| 2.6352        | 6.7   | 119662 | 2.9014          | 0.4527   |
| 2.6352        | 6.75  | 120555 | 2.9007          | 0.4533   |
| 2.6316        | 6.8   | 121448 | 2.8991          | 0.4534   |
| 2.6316        | 6.85  | 122341 | 2.8974          | 0.4528   |
| 2.6325        | 6.9   | 123234 | 2.8992          | 0.4532   |
| 2.6325        | 6.95  | 124127 | 2.8989          | 0.4533   |
| 2.6281        | 7.0   | 125020 | 2.8981          | 0.4533   |
| 2.6281        | 7.05  | 125913 | 2.8998          | 0.4532   |
| 2.6138        | 7.1   | 126806 | 2.9021          | 0.4531   |
| 2.6138        | 7.15  | 127699 | 2.9009          | 0.4535   |
| 2.6138        | 7.2   | 128592 | 2.9001          | 0.4536   |
| 2.6138        | 7.25  | 129485 | 2.8988          | 0.4536   |
| 2.6106        | 7.3   | 130378 | 2.8985          | 0.4541   |
| 2.6106        | 7.35  | 131271 | 2.8985          | 0.4538   |
| 2.6116        | 7.4   | 132164 | 2.8987          | 0.4540   |
| 2.6116        | 7.45  | 133057 | 2.8960          | 0.4537   |
| 2.6108        | 7.5   | 133950 | 2.8985          | 0.4538   |
| 2.6108        | 7.55  | 134843 | 2.8993          | 0.4536   |
| 2.6083        | 7.6   | 135736 | 2.8966          | 0.4536   |
| 2.6083        | 7.65  | 136629 | 2.8972          | 0.4539   |
| 2.6081        | 7.7   | 137522 | 2.8976          | 0.4536   |
| 2.6081        | 7.75  | 138415 | 2.8954          | 0.4537   |
| 2.6046        | 7.8   | 139308 | 2.8965          | 0.4543   |
| 2.6046        | 7.85  | 140201 | 2.8942          | 0.4540   |
| 2.6036        | 7.9   | 141094 | 2.8935          | 0.4543   |
| 2.6036        | 7.95  | 141987 | 2.8943          | 0.4543   |
| 2.602         | 8.0   | 142880 | 2.8966          | 0.4539   |
| 2.602         | 8.05  | 143773 | 2.8981          | 0.4543   |
| 2.5902        | 8.1   | 144666 | 2.8964          | 0.4541   |
| 2.5902        | 8.15  | 145559 | 2.8972          | 0.4543   |
| 2.5905        | 8.2   | 146452 | 2.8961          | 0.4543   |
| 2.5905        | 8.25  | 147345 | 2.8956          | 0.4549   |
| 2.5893        | 8.3   | 148238 | 2.8956          | 0.4548   |
| 2.5893        | 8.35  | 149131 | 2.8966          | 0.4546   |
| 2.5892        | 8.4   | 150024 | 2.8948          | 0.4546   |
| 2.5892        | 8.45  | 150917 | 2.8950          | 0.4546   |
| 2.5883        | 8.5   | 151810 | 2.8939          | 0.4549   |
| 2.5883        | 8.55  | 152703 | 2.8936          | 0.4549   |
| 2.5863        | 8.6   | 153596 | 2.8907          | 0.4551   |
| 2.5863        | 8.65  | 154489 | 2.8927          | 0.4553   |
| 2.5845        | 8.7   | 155382 | 2.8948          | 0.4551   |
| 2.5845        | 8.75  | 156275 | 2.8927          | 0.4550   |
| 2.5859        | 8.8   | 157168 | 2.8922          | 0.4556   |
| 2.5859        | 8.85  | 158061 | 2.8924          | 0.4551   |
| 2.5839        | 8.9   | 158954 | 2.8922          | 0.4555   |
| 2.5839        | 8.95  | 159847 | 2.8906          | 0.4551   |
| 2.5817        | 9.0   | 160740 | 2.8936          | 0.4555   |
| 2.5817        | 9.05  | 161633 | 2.8934          | 0.4550   |
| 2.5732        | 9.1   | 162526 | 2.8917          | 0.4552   |
| 2.5732        | 9.15  | 163419 | 2.8923          | 0.4552   |
| 2.572         | 9.2   | 164312 | 2.8926          | 0.4554   |
| 2.572         | 9.25  | 165205 | 2.8919          | 0.4553   |
| 2.5736        | 9.3   | 166098 | 2.8903          | 0.4554   |
| 2.5736        | 9.35  | 166991 | 2.8919          | 0.4556   |
| 2.5727        | 9.4   | 167884 | 2.8916          | 0.4553   |
| 2.5727        | 9.45  | 168777 | 2.8913          | 0.4554   |
| 2.5703        | 9.5   | 169670 | 2.8910          | 0.4559   |
| 2.5703        | 9.55  | 170563 | 2.8906          | 0.4555   |
| 2.568         | 9.6   | 171456 | 2.8904          | 0.4556   |
| 2.568         | 9.65  | 172349 | 2.8909          | 0.4558   |
| 2.5696        | 9.7   | 173242 | 2.8902          | 0.4560   |
| 2.5696        | 9.75  | 174135 | 2.8905          | 0.4556   |
| 2.5687        | 9.8   | 175028 | 2.8903          | 0.4557   |
| 2.5687        | 9.85  | 175921 | 2.8905          | 0.4556   |
| 2.5698        | 9.9   | 176814 | 2.8903          | 0.4558   |
| 2.5698        | 9.95  | 177707 | 2.8899          | 0.4557   |


### Framework versions

- Transformers 4.34.0
- Pytorch 2.1.0+cu121
- Datasets 2.14.5
- Tokenizers 0.14.1

{"license": "mit", "tags": ["generated_from_trainer"], "datasets": ["tyzhu/wikitext-103-raw-v1-para-permute-9"], "metrics": ["accuracy"], "base_model": "gpt2", "model-index": [{"name": "wikitext-103-raw-v1-para-permute-9_default_gpt2_from_scratch", "results": [{"task": {"type": "text-generation", "name": "Causal Language Modeling"}, "dataset": {"name": "tyzhu/wikitext-103-raw-v1-para-permute-9 default", "type": "tyzhu/wikitext-103-raw-v1-para-permute-9", "args": "default"}, "metrics": [{"type": "accuracy", "value": 0.45571032909742587, "name": "Accuracy"}]}]}]}
transformers pytorch gpt2 text-generation generated_from_trainer dataset:tyzhu/wikitext-103-raw-v1-para-permute-9 base_model:gpt2 license:mit model-index autotrain_compatible endpoints_compatible text-generation-inference region:us