---
library_name: transformers
tags:
- generated_from_trainer
datasets:
- gokulsrinivasagan/processed_book_corpus-ld-100
metrics:
- accuracy
model-index:
- name: bert_base_lda_100_v1_book
  results:
  - task:
      name: Masked Language Modeling
      type: fill-mask
    dataset:
      name: gokulsrinivasagan/processed_book_corpus-ld-100
      type: gokulsrinivasagan/processed_book_corpus-ld-100
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.7551218228545009
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# bert_base_lda_100_v1_book

This model is a fine-tuned version of [](https://huggingface.co/) on the gokulsrinivasagan/processed_book_corpus-ld-100 dataset.
It achieves the following results on the evaluation set:
- Loss: 4.4757
- Accuracy: 0.7551

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 96
- eval_batch_size: 96
- seed: 10
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- num_epochs: 25

### Training results

| Training Loss | Epoch   | Step   | Validation Loss | Accuracy |
|:-------------:|:-------:|:------:|:---------------:|:--------:|
| 9.5861        | 0.4215  | 10000  | 9.3593          | 0.1636   |
| 9.2311        | 0.8431  | 20000  | 9.0701          | 0.1641   |
| 9.1362        | 1.2646  | 30000  | 8.9889          | 0.1662   |
| 6.984         | 1.6861  | 40000  | 6.4486          | 0.4683   |
| 5.8878        | 2.1077  | 50000  | 5.5436          | 0.5941   |
| 5.6092        | 2.5292  | 60000  | 5.2862          | 0.6336   |
| 5.4567        | 2.9507  | 70000  | 5.1556          | 0.6531   |
| 5.3612        | 3.3723  | 80000  | 5.0665          | 0.6665   |
| 5.2881        | 3.7938  | 90000  | 4.9997          | 0.6763   |
| 5.2269        | 4.2153  | 100000 | 4.9488          | 0.6834   |
| 5.1854        | 4.6369  | 110000 | 4.9087          | 0.6889   |
| 5.1474        | 5.0584  | 120000 | 4.8727          | 0.6944   |
| 5.1144        | 5.4799  | 130000 | 4.8404          | 0.6990   |
| 5.0914        | 5.9014  | 140000 | 4.8134          | 0.7026   |
| 5.0579        | 6.3230  | 150000 | 4.7935          | 0.7058   |
| 5.044         | 6.7445  | 160000 | 4.7736          | 0.7089   |
| 5.0124        | 7.1660  | 170000 | 4.7533          | 0.7124   |
| 4.9997        | 7.5876  | 180000 | 4.7431          | 0.7140   |
| 4.9809        | 8.0091  | 190000 | 4.7224          | 0.7163   |
| 4.9674        | 8.4306  | 200000 | 4.7087          | 0.7189   |
| 4.9526        | 8.8522  | 210000 | 4.6976          | 0.7206   |
| 4.9422        | 9.2737  | 220000 | 4.6848          | 0.7224   |
| 4.9232        | 9.6952  | 230000 | 4.6721          | 0.7242   |
| 4.9073        | 10.1168 | 240000 | 4.6646          | 0.7261   |
| 4.9072        | 10.5383 | 250000 | 4.6502          | 0.7276   |
| 4.8916        | 10.9598 | 260000 | 4.6415          | 0.7290   |
| 4.8805        | 11.3814 | 270000 | 4.6341          | 0.7301   |
| 4.8775        | 11.8029 | 280000 | 4.6231          | 0.7318   |
| 4.86          | 12.2244 | 290000 | 4.6168          | 0.7330   |
| 4.8539        | 12.6460 | 300000 | 4.6066          | 0.7346   |
| 4.8416        | 13.0675 | 310000 | 4.5990          | 0.7359   |
| 4.8371        | 13.4890 | 320000 | 4.5932          | 0.7366   |
| 4.8312        | 13.9106 | 330000 | 4.5877          | 0.7373   |
| 4.8264        | 14.3321 | 340000 | 4.5786          | 0.7392   |
| 4.8157        | 14.7536 | 350000 | 4.5722          | 0.7398   |
| 4.8097        | 15.1751 | 360000 | 4.5686          | 0.7404   |
| 4.8045        | 15.5967 | 370000 | 4.5596          | 0.7416   |
| 4.7951        | 16.0182 | 380000 | 4.5546          | 0.7427   |
| 4.7906        | 16.4397 | 390000 | 4.5487          | 0.7434   |
| 4.7858        | 16.8613 | 400000 | 4.5452          | 0.7441   |
| 4.7798        | 17.2828 | 410000 | 4.5381          | 0.7452   |
| 4.7725        | 17.7043 | 420000 | 4.5330          | 0.7461   |
| 4.7628        | 18.1259 | 430000 | 4.5300          | 0.7466   |
| 4.7584        | 18.5474 | 440000 | 4.5251          | 0.7472   |
| 4.7573        | 18.9689 | 450000 | 4.5193          | 0.7483   |
| 4.7502        | 19.3905 | 460000 | 4.5154          | 0.7488   |
| 4.7482        | 19.8120 | 470000 | 4.5109          | 0.7496   |
| 4.738         | 20.2335 | 480000 | 4.5065          | 0.7502   |
| 4.7328        | 20.6551 | 490000 | 4.5021          | 0.7509   |
| 4.7289        | 21.0766 | 500000 | 4.4975          | 0.7515   |
| 4.7235        | 21.4981 | 510000 | 4.4950          | 0.7521   |
| 4.7191        | 21.9197 | 520000 | 4.4923          | 0.7525   |
| 4.7191        | 22.3412 | 530000 | 4.4891          | 0.7530   |
| 4.714         | 22.7627 | 540000 | 4.4877          | 0.7534   |
| 4.714         | 23.1843 | 550000 | 4.4832          | 0.7539   |
| 4.7086        | 23.6058 | 560000 | 4.4800          | 0.7544   |
| 4.7055        | 24.0273 | 570000 | 4.4785          | 0.7547   |
| 4.7024        | 24.4488 | 580000 | 4.4747          | 0.7552   |
| 4.6993        | 24.8704 | 590000 | 4.4753          | 0.7550   |


### Framework versions

- Transformers 4.46.3
- Pytorch 2.2.1+cu118
- Datasets 2.17.0
- Tokenizers 0.20.3