---
library_name: transformers
tags:
- generated_from_trainer
datasets:
- gokulsrinivasagan/processed_book_corpus-ld-100
metrics:
- accuracy
model-index:
- name: distilbert_lda_100_v1_book
  results:
  - task:
      name: Masked Language Modeling
      type: fill-mask
    dataset:
      name: gokulsrinivasagan/processed_book_corpus-ld-100
      type: gokulsrinivasagan/processed_book_corpus-ld-100
    metrics:
    - name: Accuracy
      type: accuracy
      value: 0.7269170235533601
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# distilbert_lda_100_v1_book

This model is a fine-tuned version of [](https://huggingface.co/) on the gokulsrinivasagan/processed_book_corpus-ld-100 dataset.
It achieves the following results on the evaluation set:
- Loss: 4.6471
- Accuracy: 0.7269

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 0.0001
- train_batch_size: 96
- eval_batch_size: 96
- seed: 10
- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- num_epochs: 25

### Training results

| Training Loss | Epoch   | Step   | Validation Loss | Accuracy |
|:-------------:|:-------:|:------:|:---------------:|:--------:|
| 9.8243        | 0.4215  | 10000  | 9.5492          | 0.1639   |
| 6.4155        | 0.8431  | 20000  | 6.0168          | 0.5575   |
| 5.9278        | 1.2646  | 30000  | 5.5919          | 0.6085   |
| 5.7301        | 1.6861  | 40000  | 5.4051          | 0.6304   |
| 5.5965        | 2.1077  | 50000  | 5.2807          | 0.6431   |
| 5.5012        | 2.5292  | 60000  | 5.1835          | 0.6540   |
| 5.4266        | 2.9507  | 70000  | 5.1245          | 0.6611   |
| 5.3773        | 3.3723  | 80000  | 5.0742          | 0.6676   |
| 5.3364        | 3.7938  | 90000  | 5.0321          | 0.6726   |
| 5.2973        | 4.2153  | 100000 | 5.0044          | 0.6767   |
| 5.2724        | 4.6369  | 110000 | 4.9772          | 0.6799   |
| 5.2442        | 5.0584  | 120000 | 4.9517          | 0.6836   |
| 5.2231        | 5.4799  | 130000 | 4.9291          | 0.6863   |
| 5.2074        | 5.9014  | 140000 | 4.9105          | 0.6888   |
| 5.1812        | 6.3230  | 150000 | 4.8956          | 0.6911   |
| 5.1733        | 6.7445  | 160000 | 4.8813          | 0.6934   |
| 5.147         | 7.1660  | 170000 | 4.8666          | 0.6953   |
| 5.1368        | 7.5876  | 180000 | 4.8567          | 0.6967   |
| 5.1244        | 8.0091  | 190000 | 4.8440          | 0.6982   |
| 5.1142        | 8.4306  | 200000 | 4.8315          | 0.6998   |
| 5.1017        | 8.8522  | 210000 | 4.8245          | 0.7013   |
| 5.0955        | 9.2737  | 220000 | 4.8129          | 0.7025   |
| 5.0784        | 9.6952  | 230000 | 4.8042          | 0.7039   |
| 5.0662        | 10.1168 | 240000 | 4.7974          | 0.7053   |
| 5.067         | 10.5383 | 250000 | 4.7871          | 0.7062   |
| 5.0545        | 10.9598 | 260000 | 4.7792          | 0.7074   |
| 5.0461        | 11.3814 | 270000 | 4.7762          | 0.7082   |
| 5.0456        | 11.8029 | 280000 | 4.7663          | 0.7093   |
| 5.0294        | 12.2244 | 290000 | 4.7599          | 0.7103   |
| 5.0258        | 12.6460 | 300000 | 4.7528          | 0.7113   |
| 5.0149        | 13.0675 | 310000 | 4.7464          | 0.7123   |
| 5.0114        | 13.4890 | 320000 | 4.7420          | 0.7131   |
| 5.0086        | 13.9106 | 330000 | 4.7378          | 0.7137   |
| 5.004         | 14.3321 | 340000 | 4.7310          | 0.7147   |
| 4.9941        | 14.7536 | 350000 | 4.7263          | 0.7152   |
| 4.9902        | 15.1751 | 360000 | 4.7222          | 0.7157   |
| 4.9867        | 15.5967 | 370000 | 4.7158          | 0.7168   |
| 4.9796        | 16.0182 | 380000 | 4.7116          | 0.7175   |
| 4.9751        | 16.4397 | 390000 | 4.7051          | 0.7180   |
| 4.9683        | 16.8613 | 400000 | 4.7038          | 0.7184   |
| 4.967         | 17.2828 | 410000 | 4.6955          | 0.7196   |
| 4.961         | 17.7043 | 420000 | 4.6947          | 0.7200   |
| 4.953         | 18.1259 | 430000 | 4.6910          | 0.7204   |
| 4.9491        | 18.5474 | 440000 | 4.6884          | 0.7208   |
| 4.9485        | 18.9689 | 450000 | 4.6825          | 0.7217   |
| 4.9439        | 19.3905 | 460000 | 4.6790          | 0.7222   |
| 4.9417        | 19.8120 | 470000 | 4.6757          | 0.7226   |
| 4.9334        | 20.2335 | 480000 | 4.6713          | 0.7233   |
| 4.929         | 20.6551 | 490000 | 4.6686          | 0.7238   |
| 4.925         | 21.0766 | 500000 | 4.6645          | 0.7242   |
| 4.9207        | 21.4981 | 510000 | 4.6618          | 0.7246   |
| 4.9177        | 21.9197 | 520000 | 4.6599          | 0.7250   |
| 4.9191        | 22.3412 | 530000 | 4.6584          | 0.7252   |
| 4.9138        | 22.7627 | 540000 | 4.6577          | 0.7255   |
| 4.9139        | 23.1843 | 550000 | 4.6533          | 0.7259   |
| 4.9098        | 23.6058 | 560000 | 4.6508          | 0.7264   |
| 4.9063        | 24.0273 | 570000 | 4.6497          | 0.7265   |
| 4.9048        | 24.4488 | 580000 | 4.6457          | 0.7271   |
| 4.9011        | 24.8704 | 590000 | 4.6463          | 0.7270   |


### Framework versions

- Transformers 4.46.3
- Pytorch 2.2.1+cu118
- Datasets 2.17.0
- Tokenizers 0.20.3