




# mekjr1/guilbert-base-uncased

This model is a fine-tuned version of [bert-base-uncased](https://huggingface.co/bert-base-uncased)  on an guilbert dataset. It is a masked language model that predicts missing tokens in a sentence.

## Model description

The model is based on the `bert-base-uncased` architecture, which has 12 layers, 768 hidden units, and 12 attention heads. It has been fine-tuned on a dataset with samples labeled as guilt or non-guilt from the Vent dataset. The model was trained with a maximum sequence length of 128 tokens and a batch size of 32. The training process used the AdamW optimizer with a learning rate of 2e-5, a weight decay rate of 0.01, and a linear learning rate warmup over 1,000 steps. The model achieved a validation loss of 1.8529 after 8 epochs.

## Intended uses & limitations

This model can be used for predicting missing tokens in text sequences, particularly in the context of detecting guilt emotion in documents or other relevant applications. 
However, the accuracy of the model may be limited by the quality and representativeness of the training data, as well as the biases present in the pre-trained `bert-base-uncased` architecture.

## Training and evaluation data

The model was trained on a dataset of samples labeled as guilt or non-guilt from the guilbert dataset (Extracted from Vent).

## Training procedure

The model was trained using TensorFlow Keras with the AdamW optimizer and a learning rate of 2e-5. The training process used a batch size of 32 and a maximum sequence length of 128 tokens. The optimizer used a weight decay rate of 0.01 and a linear learning rate warmup over 1,000 steps. The model was trained for 8 epochs, with early stopping based on the validation loss. The training process achieved a validation loss of 1.8529 after 8 epochs.

### Training hyperparameters

The following hyperparameters were used during training:

- Optimizer: `AdamWeightDecay` with a learning rate of `WarmUp(initial_learning_rate=2e-05, decay_schedule_fn=PolynomialDecay(initial_learning_rate=2e-05, decay_steps=7167, end_learning_rate=0.0, power=1.0, cycle=False), warmup_steps=1000, power=1.0)`
- Weight decay rate: 0.01
- Batch size: 32
- Maximum sequence length: 128
- Number of warmup steps: 1,000
- Number of training steps: 1,761
The following hyperparameters were used during training:

- optimizer: {'name': 'AdamWeightDecay', 'learning_rate': {'class_name': 'WarmUp', 'config': {'initial_learning_rate': 2e-05, 'decay_schedule_fn': {'class_name': 'PolynomialDecay', 'config': {'initial_learning_rate': 2e-05, 'decay_steps': 7167, 'end_learning_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, '__passive_serialization__': True}, 'warmup_steps': 1000, 'power': 1.0, 'name': None}}, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False, 'weight_decay_rate': 0.01}
- training_precision: mixed_float16

### Training results

The following table shows the training and validation loss for each epoch:

| Train Loss | Validation Loss | Epoch |
|:----------:|:---------------:|:-----:|
| 2.0976     | 1.8593          | 0     |
| 1.9643     | 1.8547          | 1     |
| 1.9651     | 1.9003          | 2     |
| 1.9608     | 1.8617          | 3     |
| 1.9646     | 1.8756          | 4     |
| 1.9626     | 1.9024          | 5     |
| 1.9574     | 1.8421          | 6     |
| 1.9594     | 1.8632          | 7     |
| 1.9616     | 1.8529          | 8     |



### Framework versions

- Transformers 4.26.1
- TensorFlow 2.11.0
- Datasets 2.10.1
- Tokenizers 0.13.2
{"language": ["en"], "license": "apache-2.0", "tags": ["generated_from_keras_callback"], "datasets": ["mekjr1/guilbert_lm"], "model-index": [{"name": "mekjr1/guilbert-base-uncased", "results": []}]}
transformers tf bert fill-mask generated_from_keras_callback en dataset:mekjr1/guilbert_lm license:apache-2.0 autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #tf #bert #fill-mask #generated_from_keras_callback #en #dataset-mekjr1/guilbert_lm #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 

mekjr1/guilbert-base-uncased
============================


This model is a fine-tuned version of bert-base-uncased on an guilbert dataset. It is a masked language model that predicts missing tokens in a sentence.


Model description
-----------------


The model is based on the 'bert-base-uncased' architecture, which has 12 layers, 768 hidden units, and 12 attention heads. It has been fine-tuned on a dataset with samples labeled as guilt or non-guilt from the Vent dataset. The model was trained with a maximum sequence length of 128 tokens and a batch size of 32. The training process used the AdamW optimizer with a learning rate of 2e-5, a weight decay rate of 0.01, and a linear learning rate warmup over 1,000 steps. The model achieved a validation loss of 1.8529 after 8 epochs.


Intended uses & limitations
---------------------------


This model can be used for predicting missing tokens in text sequences, particularly in the context of detecting guilt emotion in documents or other relevant applications.
However, the accuracy of the model may be limited by the quality and representativeness of the training data, as well as the biases present in the pre-trained 'bert-base-uncased' architecture.


Training and evaluation data
----------------------------


The model was trained on a dataset of samples labeled as guilt or non-guilt from the guilbert dataset (Extracted from Vent).


Training procedure
------------------


The model was trained using TensorFlow Keras with the AdamW optimizer and a learning rate of 2e-5. The training process used a batch size of 32 and a maximum sequence length of 128 tokens. The optimizer used a weight decay rate of 0.01 and a linear learning rate warmup over 1,000 steps. The model was trained for 8 epochs, with early stopping based on the validation loss. The training process achieved a validation loss of 1.8529 after 8 epochs.


### Training hyperparameters


The following hyperparameters were used during training:


* Optimizer: 'AdamWeightDecay' with a learning rate of 'WarmUp(initial\_learning\_rate=2e-05, decay\_schedule\_fn=PolynomialDecay(initial\_learning\_rate=2e-05, decay\_steps=7167, end\_learning\_rate=0.0, power=1.0, cycle=False), warmup\_steps=1000, power=1.0)'
* Weight decay rate: 0.01
* Batch size: 32
* Maximum sequence length: 128
* Number of warmup steps: 1,000
* Number of training steps: 1,761
The following hyperparameters were used during training:
* optimizer: {'name': 'AdamWeightDecay', 'learning\_rate': {'class\_name': 'WarmUp', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_schedule\_fn': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 7167, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, '**passive\_serialization**': True}, 'warmup\_steps': 1000, 'power': 1.0, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False, 'weight\_decay\_rate': 0.01}
* training\_precision: mixed\_float16


### Training results


The following table shows the training and validation loss for each epoch:



### Framework versions


* Transformers 4.26.1
* TensorFlow 2.11.0
* Datasets 2.10.1
* Tokenizers 0.13.2



### Training hyperparameters


The following hyperparameters were used during training:


* Optimizer: 'AdamWeightDecay' with a learning rate of 'WarmUp(initial\_learning\_rate=2e-05, decay\_schedule\_fn=PolynomialDecay(initial\_learning\_rate=2e-05, decay\_steps=7167, end\_learning\_rate=0.0, power=1.0, cycle=False), warmup\_steps=1000, power=1.0)'
* Weight decay rate: 0.01
* Batch size: 32
* Maximum sequence length: 128
* Number of warmup steps: 1,000
* Number of training steps: 1,761
The following hyperparameters were used during training:
* optimizer: {'name': 'AdamWeightDecay', 'learning\_rate': {'class\_name': 'WarmUp', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_schedule\_fn': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 7167, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, '**passive\_serialization**': True}, 'warmup\_steps': 1000, 'power': 1.0, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False, 'weight\_decay\_rate': 0.01}
* training\_precision: mixed\_float16 ### Training results


The following table shows the training and validation loss for each epoch: ### Framework versions


* Transformers 4.26.1
* TensorFlow 2.11.0
* Datasets 2.10.1
* Tokenizers 0.13.2
TAGS
#transformers #tf #bert #fill-mask #generated_from_keras_callback #en #dataset-mekjr1/guilbert_lm #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
 ### Training hyperparameters


The following hyperparameters were used during training:


* Optimizer: 'AdamWeightDecay' with a learning rate of 'WarmUp(initial\_learning\_rate=2e-05, decay\_schedule\_fn=PolynomialDecay(initial\_learning\_rate=2e-05, decay\_steps=7167, end\_learning\_rate=0.0, power=1.0, cycle=False), warmup\_steps=1000, power=1.0)'
* Weight decay rate: 0.01
* Batch size: 32
* Maximum sequence length: 128
* Number of warmup steps: 1,000
* Number of training steps: 1,761
The following hyperparameters were used during training:
* optimizer: {'name': 'AdamWeightDecay', 'learning\_rate': {'class\_name': 'WarmUp', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_schedule\_fn': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 7167, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, '**passive\_serialization**': True}, 'warmup\_steps': 1000, 'power': 1.0, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False, 'weight\_decay\_rate': 0.01}
* training\_precision: mixed\_float16 ### Training results


The following table shows the training and validation loss for each epoch: ### Framework versions


* Transformers 4.26.1
* TensorFlow 2.11.0
* Datasets 2.10.1
* Tokenizers 0.13.2
TAGS
#transformers #tf #bert #fill-mask #generated_from_keras_callback #en #dataset-mekjr1/guilbert_lm #license-apache-2.0 #autotrain_compatible #endpoints_compatible #region-us 
### Training hyperparameters


The following hyperparameters were used during training:


* Optimizer: 'AdamWeightDecay' with a learning rate of 'WarmUp(initial\_learning\_rate=2e-05, decay\_schedule\_fn=PolynomialDecay(initial\_learning\_rate=2e-05, decay\_steps=7167, end\_learning\_rate=0.0, power=1.0, cycle=False), warmup\_steps=1000, power=1.0)'
* Weight decay rate: 0.01
* Batch size: 32
* Maximum sequence length: 128
* Number of warmup steps: 1,000
* Number of training steps: 1,761
The following hyperparameters were used during training:
* optimizer: {'name': 'AdamWeightDecay', 'learning\_rate': {'class\_name': 'WarmUp', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_schedule\_fn': {'class\_name': 'PolynomialDecay', 'config': {'initial\_learning\_rate': 2e-05, 'decay\_steps': 7167, 'end\_learning\_rate': 0.0, 'power': 1.0, 'cycle': False, 'name': None}, '**passive\_serialization**': True}, 'warmup\_steps': 1000, 'power': 1.0, 'name': None}}, 'decay': 0.0, 'beta\_1': 0.9, 'beta\_2': 0.999, 'epsilon': 1e-08, 'amsgrad': False, 'weight\_decay\_rate': 0.01}
* training\_precision: mixed\_float16### Training results


The following table shows the training and validation loss for each epoch:### Framework versions


* Transformers 4.26.1
* TensorFlow 2.11.0
* Datasets 2.10.1
* Tokenizers 0.13.2