peft pytorch btlm generated_from_trainer custom_code dataset:iarfmoose/question_generator dataset:Defalt-404/Bittensor_validator dataset:multi_news dataset:cnn_dailymail base_model:cerebras/btlm-3b-8k-base license:apache-2.0 region:us
---
license: apache-2.0
base_model: cerebras/btlm-3b-8k-base
tags:
- generated_from_trainer
model-index:
- name: app/bt_qa-out
  results: []
datasets:
- iarfmoose/question_generator
- Defalt-404/Bittensor_validator
- multi_news
- cnn_dailymail
library_name: peft
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
# app/bt_qa-out

This model is a fine-tuned version of [cerebras/btlm-3b-8k-base](https://huggingface.co/cerebras/btlm-3b-8k-base) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 1.7451

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 8.5e-05
- train_batch_size: 3
- eval_batch_size: 3
- seed: 42
- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 32
- num_epochs: 1

### Training results

| Training Loss | Epoch | Step   | Validation Loss |
|:-------------:|:-----:|:------:|:---------------:|
| 2.7377        | 0.0   | 500    | 2.5700          |
| 2.5689        | 0.01  | 1000   | 2.4172          |
| 2.6676        | 0.01  | 1500   | 2.3622          |
| 2.4234        | 0.01  | 2000   | 2.3293          |
| 2.2221        | 0.01  | 2500   | 2.3090          |
| 2.3873        | 0.02  | 3000   | 2.2940          |
| 2.2688        | 0.02  | 3500   | 2.2644          |
| 2.4194        | 0.02  | 4000   | 2.2725          |
| 2.4086        | 0.03  | 4500   | 2.2561          |
| 2.4476        | 0.03  | 5000   | 2.2477          |
| 2.1512        | 0.03  | 5500   | 2.2330          |
| 2.1428        | 0.03  | 6000   | 2.2235          |
| 2.2834        | 0.04  | 6500   | 2.2141          |
| 2.2918        | 0.04  | 7000   | 2.2124          |
| 2.4352        | 0.04  | 7500   | 2.2074          |
| 1.7196        | 0.05  | 8000   | 2.2038          |
| 2.2394        | 0.05  | 8500   | 2.1973          |
| 2.1632        | 0.05  | 9000   | 2.1856          |
| 2.4313        | 0.06  | 9500   | 2.1820          |
| 2.4584        | 0.06  | 10000  | 2.1764          |
| 2.3359        | 0.06  | 10500  | 2.1780          |
| 2.2105        | 0.06  | 11000  | 2.1671          |
| 2.3152        | 0.07  | 11500  | 2.1603          |
| 2.3012        | 0.07  | 12000  | 2.1572          |
| 2.4636        | 0.07  | 12500  | 2.1553          |
| 2.0974        | 0.08  | 13000  | 2.1511          |
| 2.298         | 0.08  | 13500  | 2.1481          |
| 2.3312        | 0.08  | 14000  | 2.1445          |
| 2.5315        | 0.08  | 14500  | 2.1381          |
| 2.1854        | 0.09  | 15000  | 2.1364          |
| 2.3069        | 0.09  | 15500  | 2.1355          |
| 2.0756        | 0.09  | 16000  | 2.1331          |
| 2.0094        | 0.1   | 16500  | 2.1306          |
| 2.2674        | 0.1   | 17000  | 2.1230          |
| 1.8427        | 0.1   | 17500  | 2.1176          |
| 2.2277        | 0.1   | 18000  | 2.1168          |
| 2.1398        | 0.11  | 18500  | 2.1152          |
| 1.9927        | 0.11  | 19000  | 2.1088          |
| 2.0119        | 0.11  | 19500  | 2.1105          |
| 2.5796        | 0.12  | 20000  | 2.1040          |
| 1.3256        | 0.12  | 20500  | 2.0993          |
| 2.2051        | 0.12  | 21000  | 2.0992          |
| 1.628         | 0.13  | 21500  | 2.0944          |
| 2.1926        | 0.13  | 22000  | 2.0927          |
| 1.6482        | 0.13  | 22500  | 2.0873          |
| 2.1122        | 0.13  | 23000  | 2.0830          |
| 1.7405        | 0.14  | 23500  | 2.0828          |
| 2.2685        | 0.14  | 24000  | 2.0784          |
| 2.1062        | 0.14  | 24500  | 2.0766          |
| 2.1308        | 0.15  | 25000  | 2.0714          |
| 1.9122        | 0.15  | 25500  | 2.0719          |
| 2.3549        | 0.15  | 26000  | 2.0643          |
| 2.2159        | 0.15  | 26500  | 2.0655          |
| 1.493         | 0.16  | 27000  | 2.0598          |
| 1.893         | 0.16  | 27500  | 2.0557          |
| 2.1902        | 0.16  | 28000  | 2.0533          |
| 2.2353        | 0.17  | 28500  | 2.0524          |
| 1.8736        | 0.17  | 29000  | 2.0519          |
| 2.0511        | 0.17  | 29500  | 2.0449          |
| 1.2872        | 0.17  | 30000  | 2.0453          |
| 1.6353        | 0.18  | 30500  | 2.0377          |
| 1.992         | 0.18  | 31000  | 2.0419          |
| 2.3586        | 0.18  | 31500  | 2.0353          |
| 1.9453        | 0.19  | 32000  | 2.0330          |
| 2.1322        | 0.19  | 32500  | 2.0305          |
| 2.2887        | 0.19  | 33000  | 2.0253          |
| 2.0268        | 0.2   | 33500  | 2.0267          |
| 1.8397        | 0.2   | 34000  | 2.0207          |
| 2.5165        | 0.2   | 34500  | 2.0202          |
| 1.9142        | 0.2   | 35000  | 2.0139          |
| 1.5993        | 0.21  | 35500  | 2.0179          |
| 2.1691        | 0.21  | 36000  | 2.0102          |
| 2.4948        | 0.21  | 36500  | 2.0089          |
| 1.5422        | 0.22  | 37000  | 2.0039          |
| 1.4566        | 0.22  | 37500  | 2.0014          |
| 1.852         | 0.22  | 38000  | 2.0043          |
| 2.199         | 0.22  | 38500  | 1.9987          |
| 1.4852        | 0.23  | 39000  | 1.9976          |
| 1.3           | 0.23  | 39500  | 1.9936          |
| 2.1237        | 0.23  | 40000  | 1.9917          |
| 1.691         | 0.24  | 40500  | 1.9887          |
| 2.2169        | 0.24  | 41000  | 1.9870          |
| 2.1991        | 0.24  | 41500  | 1.9851          |
| 1.9517        | 0.24  | 42000  | 1.9806          |
| 1.6369        | 0.25  | 42500  | 1.9762          |
| 2.2759        | 0.25  | 43000  | 1.9753          |
| 2.2923        | 0.25  | 43500  | 1.9748          |
| 2.2552        | 0.26  | 44000  | 1.9702          |
| 2.066         | 0.26  | 44500  | 1.9683          |
| 2.2703        | 0.26  | 45000  | 1.9686          |
| 2.3544        | 0.27  | 45500  | 1.9648          |
| 2.255         | 0.27  | 46000  | 1.9635          |
| 1.8732        | 0.27  | 46500  | 1.9639          |
| 2.1203        | 0.27  | 47000  | 1.9590          |
| 2.1314        | 0.28  | 47500  | 1.9573          |
| 1.8511        | 0.28  | 48000  | 1.9533          |
| 2.1471        | 0.28  | 48500  | 1.9514          |
| 1.8417        | 0.29  | 49000  | 1.9509          |
| 2.4485        | 0.29  | 49500  | 1.9502          |
| 2.0708        | 0.29  | 50000  | 1.9455          |
| 1.8272        | 0.29  | 50500  | 1.9416          |
| 1.6232        | 0.3   | 51000  | 1.9380          |
| 1.6785        | 0.3   | 51500  | 1.9358          |
| 1.5734        | 0.3   | 52000  | 1.9313          |
| 1.9737        | 0.31  | 52500  | 1.9301          |
| 1.8393        | 0.31  | 53000  | 1.9295          |
| 1.4789        | 0.31  | 53500  | 1.9281          |
| 2.2062        | 0.31  | 54000  | 1.9273          |
| 2.3501        | 0.32  | 54500  | 1.9236          |
| 2.2756        | 0.32  | 55000  | 1.9218          |
| 2.1001        | 0.32  | 55500  | 1.9215          |
| 2.0342        | 0.33  | 56000  | 1.9179          |
| 1.8066        | 0.33  | 56500  | 1.9143          |
| 1.8322        | 0.33  | 57000  | 1.9137          |
| 2.0926        | 0.34  | 57500  | 1.9106          |
| 2.2106        | 0.34  | 58000  | 1.9083          |
| 2.0666        | 0.34  | 58500  | 1.9055          |
| 2.2082        | 0.34  | 59000  | 1.9026          |
| 2.1768        | 0.35  | 59500  | 1.9007          |
| 1.7091        | 0.35  | 60000  | 1.8967          |
| 1.7585        | 0.35  | 60500  | 1.8946          |
| 1.8968        | 0.36  | 61000  | 1.8936          |
| 2.107         | 0.36  | 61500  | 1.8906          |
| 1.5162        | 0.36  | 62000  | 1.8870          |
| 2.0642        | 0.36  | 62500  | 1.8836          |
| 2.0399        | 0.37  | 63000  | 1.8813          |
| 2.3971        | 0.37  | 63500  | 1.8785          |
| 1.7433        | 0.37  | 64000  | 1.8797          |
| 2.0971        | 0.38  | 64500  | 1.8743          |
| 1.8212        | 0.38  | 65000  | 1.8726          |
| 2.1023        | 0.38  | 65500  | 1.8695          |
| 1.9735        | 0.38  | 66000  | 1.8674          |
| 1.3196        | 0.39  | 66500  | 1.8657          |
| 1.9825        | 0.39  | 67000  | 1.8629          |
| 2.0356        | 0.39  | 67500  | 1.8604          |
| 1.8522        | 0.4   | 68000  | 1.8581          |
| 2.2666        | 0.4   | 68500  | 1.8568          |
| 2.3575        | 0.4   | 69000  | 1.8538          |
| 2.0086        | 0.41  | 69500  | 1.8537          |
| 1.9811        | 0.41  | 70000  | 1.8512          |
| 2.0702        | 0.41  | 70500  | 1.8485          |
| 1.8554        | 0.41  | 71000  | 1.8456          |
| 0.5356        | 0.42  | 71500  | 1.8437          |
| 1.4742        | 0.42  | 72000  | 1.8413          |
| 2.1901        | 0.42  | 72500  | 1.8420          |
| 1.7868        | 0.43  | 73000  | 1.8383          |
| 1.3144        | 0.43  | 73500  | 1.8371          |
| 2.1158        | 0.43  | 74000  | 1.8347          |
| 2.0779        | 0.43  | 74500  | 1.8331          |
| 1.9756        | 0.44  | 75000  | 1.8323          |
| 2.3395        | 0.44  | 75500  | 1.8309          |
| 1.895         | 0.44  | 76000  | 1.8283          |
| 2.0369        | 0.45  | 76500  | 1.8274          |
| 1.8068        | 0.45  | 77000  | 1.8251          |
| 2.2153        | 0.45  | 77500  | 1.8227          |
| 2.1389        | 0.45  | 78000  | 1.8212          |
| 1.9166        | 0.46  | 78500  | 1.8197          |
| 1.711         | 0.46  | 79000  | 1.8187          |
| 1.9102        | 0.46  | 79500  | 1.8165          |
| 0.8358        | 0.47  | 80000  | 1.8163          |
| 1.7278        | 0.47  | 80500  | 1.8148          |
| 1.601         | 0.47  | 81000  | 1.8126          |
| 1.9794        | 0.48  | 81500  | 1.8107          |
| 1.7323        | 0.48  | 82000  | 1.8095          |
| 2.2911        | 0.48  | 82500  | 1.8090          |
| 1.8962        | 0.48  | 83000  | 1.8065          |
| 2.3055        | 0.49  | 83500  | 1.8052          |
| 1.6899        | 0.49  | 84000  | 1.8037          |
| 1.6409        | 0.49  | 84500  | 1.8031          |
| 1.9116        | 0.5   | 85000  | 1.8011          |
| 0.6875        | 0.5   | 85500  | 1.8003          |
| 2.0829        | 0.5   | 86000  | 1.7983          |
| 1.5716        | 0.5   | 86500  | 1.7981          |
| 2.4537        | 0.51  | 87000  | 1.7961          |
| 1.8236        | 0.51  | 87500  | 1.7942          |
| 1.641         | 0.51  | 88000  | 1.7931          |
| 1.5533        | 0.52  | 88500  | 1.7916          |
| 1.679         | 0.52  | 89000  | 1.7902          |
| 2.1463        | 0.52  | 89500  | 1.7893          |
| 1.5477        | 0.52  | 90000  | 1.7884          |
| 1.2346        | 0.53  | 90500  | 1.7873          |
| 1.3352        | 0.53  | 91000  | 1.7859          |
| 2.1039        | 0.53  | 91500  | 1.7850          |
| 2.0818        | 0.54  | 92000  | 1.7834          |
| 1.3987        | 0.54  | 92500  | 1.7830          |
| 1.4544        | 0.54  | 93000  | 1.7827          |
| 0.4043        | 0.55  | 93500  | 1.7811          |
| 2.0149        | 0.55  | 94000  | 1.7794          |
| 1.9845        | 0.55  | 94500  | 1.7789          |
| 2.1053        | 0.55  | 95000  | 1.7775          |
| 2.1572        | 0.56  | 95500  | 1.7768          |
| 2.0754        | 0.56  | 96000  | 1.7761          |
| 1.7675        | 0.56  | 96500  | 1.7754          |
| 2.0023        | 0.57  | 97000  | 1.7743          |
| 1.2653        | 0.57  | 97500  | 1.7736          |
| 1.5566        | 0.57  | 98000  | 1.7728          |
| 1.9408        | 0.57  | 98500  | 1.7724          |
| 2.0936        | 0.58  | 99000  | 1.7713          |
| 0.5687        | 0.58  | 99500  | 1.7706          |
| 2.2833        | 0.58  | 100000 | 1.7702          |
| 1.6689        | 0.59  | 100500 | 1.7690          |
| 1.5198        | 0.59  | 101000 | 1.7684          |
| 1.6968        | 0.59  | 101500 | 1.7679          |
| 2.2034        | 0.59  | 102000 | 1.7674          |
| 1.7902        | 0.6   | 102500 | 1.7665          |
| 2.0557        | 0.6   | 103000 | 1.7658          |
| 1.8617        | 0.6   | 103500 | 1.7650          |
| 1.8749        | 0.61  | 104000 | 1.7637          |
| 1.7674        | 0.61  | 104500 | 1.7632          |
| 1.4269        | 0.61  | 105000 | 1.7627          |
| 1.989         | 0.62  | 105500 | 1.7621          |
| 2.1026        | 0.62  | 106000 | 1.7615          |
| 2.0304        | 0.62  | 106500 | 1.7609          |
| 1.6286        | 0.62  | 107000 | 1.7603          |
| 0.9544        | 0.63  | 107500 | 1.7599          |
| 1.6421        | 0.63  | 108000 | 1.7588          |
| 1.9841        | 0.63  | 108500 | 1.7586          |
| 1.7453        | 0.64  | 109000 | 1.7581          |
| 1.2119        | 0.64  | 109500 | 1.7575          |
| 2.1092        | 0.64  | 110000 | 1.7568          |
| 2.0849        | 0.64  | 110500 | 1.7564          |
| 1.9162        | 0.65  | 111000 | 1.7562          |
| 1.01          | 0.65  | 111500 | 1.7560          |
| 1.301         | 0.65  | 112000 | 1.7556          |
| 0.315         | 0.66  | 112500 | 1.7552          |
| 1.9964        | 0.66  | 113000 | 1.7548          |
| 2.4035        | 0.66  | 113500 | 1.7544          |
| 1.3559        | 0.66  | 114000 | 1.7542          |
| 2.1874        | 0.67  | 114500 | 1.7538          |
| 1.4373        | 0.67  | 115000 | 1.7534          |
| 0.0639        | 0.67  | 115500 | 1.7529          |
| 1.7667        | 0.68  | 116000 | 1.7526          |
| 1.6204        | 0.68  | 116500 | 1.7524          |
| 1.9859        | 0.68  | 117000 | 1.7521          |
| 0.9717        | 0.69  | 117500 | 1.7516          |
| 1.8844        | 0.69  | 118000 | 1.7514          |
| 1.3336        | 0.69  | 118500 | 1.7509          |
| 1.5781        | 0.69  | 119000 | 1.7506          |
| 1.8449        | 0.7   | 119500 | 1.7505          |
| 1.5305        | 0.7   | 120000 | 1.7503          |
| 2.1904        | 0.7   | 120500 | 1.7500          |
| 2.2285        | 0.71  | 121000 | 1.7496          |
| 1.8097        | 0.71  | 121500 | 1.7494          |
| 2.3631        | 0.71  | 122000 | 1.7493          |
| 2.0893        | 0.71  | 122500 | 1.7491          |
| 2.1201        | 0.72  | 123000 | 1.7489          |
| 1.8334        | 0.72  | 123500 | 1.7488          |
| 2.0222        | 0.72  | 124000 | 1.7486          |
| 1.6339        | 0.73  | 124500 | 1.7484          |
| 1.6754        | 0.73  | 125000 | 1.7482          |
| 1.3973        | 0.73  | 125500 | 1.7480          |
| 2.0594        | 0.73  | 126000 | 1.7479          |
| 1.8674        | 0.74  | 126500 | 1.7478          |
| 2.1948        | 0.74  | 127000 | 1.7476          |
| 1.4148        | 0.74  | 127500 | 1.7475          |
| 1.6734        | 0.75  | 128000 | 1.7473          |
| 2.2787        | 0.75  | 128500 | 1.7472          |
| 1.8999        | 0.75  | 129000 | 1.7471          |
| 1.6945        | 0.76  | 129500 | 1.7470          |
| 2.0165        | 0.76  | 130000 | 1.7469          |
| 2.2232        | 0.76  | 130500 | 1.7468          |
| 1.6201        | 0.76  | 131000 | 1.7466          |
| 2.4878        | 0.77  | 131500 | 1.7465          |
| 1.5317        | 0.77  | 132000 | 1.7465          |
| 1.9361        | 0.77  | 132500 | 1.7464          |
| 1.7127        | 0.78  | 133000 | 1.7463          |
| 1.7045        | 0.78  | 133500 | 1.7462          |
| 2.1827        | 0.78  | 134000 | 1.7461          |
| 2.0534        | 0.78  | 134500 | 1.7461          |
| 2.0808        | 0.79  | 135000 | 1.7460          |
| 1.9572        | 0.79  | 135500 | 1.7459          |
| 1.8762        | 0.79  | 136000 | 1.7459          |
| 1.4686        | 0.8   | 136500 | 1.7458          |
| 1.6241        | 0.8   | 137000 | 1.7458          |
| 1.4219        | 0.8   | 137500 | 1.7457          |
| 2.1605        | 0.8   | 138000 | 1.7457          |
| 2.1298        | 0.81  | 138500 | 1.7456          |
| 1.414         | 0.81  | 139000 | 1.7456          |
| 1.0115        | 0.81  | 139500 | 1.7455          |
| 1.9471        | 0.82  | 140000 | 1.7455          |
| 1.8873        | 0.82  | 140500 | 1.7455          |
| 1.8286        | 0.82  | 141000 | 1.7454          |
| 2.1418        | 0.83  | 141500 | 1.7454          |
| 1.9755        | 0.83  | 142000 | 1.7454          |
| 1.6908        | 0.83  | 142500 | 1.7454          |
| 2.3842        | 0.83  | 143000 | 1.7453          |
| 1.7665        | 0.84  | 143500 | 1.7453          |
| 1.8266        | 0.84  | 144000 | 1.7453          |
| 0.8768        | 0.84  | 144500 | 1.7453          |
| 1.2274        | 0.85  | 145000 | 1.7453          |
| 1.6647        | 0.85  | 145500 | 1.7453          |
| 1.4071        | 0.85  | 146000 | 1.7452          |
| 1.6073        | 0.85  | 146500 | 1.7452          |
| 2.201         | 0.86  | 147000 | 1.7452          |
| 1.5504        | 0.86  | 147500 | 1.7452          |
| 1.4377        | 0.86  | 148000 | 1.7452          |
| 1.4453        | 0.87  | 148500 | 1.7452          |
| 1.6929        | 0.87  | 149000 | 1.7451          |
| 1.7631        | 0.87  | 149500 | 1.7451          |
| 2.0868        | 0.87  | 150000 | 1.7451          |
| 0.6434        | 0.88  | 150500 | 1.7451          |
| 1.4851        | 0.88  | 151000 | 1.7451          |
| 1.5365        | 0.88  | 151500 | 1.7451          |
| 1.8129        | 0.89  | 152000 | 1.7451          |
| 1.1623        | 0.89  | 152500 | 1.7451          |
| 2.0714        | 0.89  | 153000 | 1.7451          |
| 1.9363        | 0.9   | 153500 | 1.7451          |
| 1.6408        | 0.9   | 154000 | 1.7451          |
| 0.618         | 0.9   | 154500 | 1.7451          |
| 1.7957        | 0.9   | 155000 | 1.7451          |
| 2.0056        | 0.91  | 155500 | 1.7451          |
| 1.3893        | 0.91  | 156000 | 1.7451          |
| 2.1426        | 0.91  | 156500 | 1.7451          |
| 1.6766        | 0.92  | 157000 | 1.7451          |
| 1.4206        | 0.92  | 157500 | 1.7451          |
| 1.7285        | 0.92  | 158000 | 1.7451          |
| 1.5779        | 0.92  | 158500 | 1.7451          |
| 1.8675        | 0.93  | 159000 | 1.7451          |
| 2.0217        | 0.93  | 159500 | 1.7451          |
| 0.9516        | 0.93  | 160000 | 1.7451          |
| 2.219         | 0.94  | 160500 | 1.7450          |
| 1.6214        | 0.94  | 161000 | 1.7451          |
| 1.7134        | 0.94  | 161500 | 1.7451          |
| 1.6128        | 0.94  | 162000 | 1.7451          |
| 2.0817        | 0.95  | 162500 | 1.7450          |
| 1.8055        | 0.95  | 163000 | 1.7451          |
| 1.909         | 0.95  | 163500 | 1.7451          |
| 1.7844        | 0.96  | 164000 | 1.7451          |
| 2.0719        | 0.96  | 164500 | 1.7451          |
| 1.8698        | 0.96  | 165000 | 1.7451          |
| 1.6926        | 0.96  | 165500 | 1.7451          |
| 2.2161        | 0.97  | 166000 | 1.7451          |
| 2.1111        | 0.97  | 166500 | 1.7451          |
| 1.8004        | 0.97  | 167000 | 1.7451          |
| 2.2364        | 0.98  | 167500 | 1.7451          |
| 1.6716        | 0.98  | 168000 | 1.7451          |
| 2.1804        | 0.98  | 168500 | 1.7451          |
| 1.2691        | 0.99  | 169000 | 1.7451          |
| 1.8306        | 0.99  | 169500 | 1.7451          |
| 0.5662        | 0.99  | 170000 | 1.7451          |
| 1.6516        | 0.99  | 170500 | 1.7451          |
| 2.0576        | 1.0   | 171000 | 1.7451          |
| 1.3638        | 1.0   | 171500 | 1.7451          |


### Framework versions

- Transformers 4.34.1
- Pytorch 2.0.1+cu117
- Datasets 2.14.6
- Tokenizers 0.14.1
