transformers pytorch bert fill-mask generated_from_trainer autotrain_compatible endpoints_compatible region:us
---
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: MLM-final-scraped-g
  results: []
---

# potocatalysisbert

This model is pretrained on a corpus of papers on photcatalysis. For more detailed training procedures, see
"How beneficial is pre-training on a narrow domain-specific corpus for information extraction about photocatalytic water splitting?" by Taketomo Isazawa and Jacqueline M. Cole.

It achieves the following results on the evaluation set:
- Loss: 1.0215
- Accuracy: 0.7554

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 8e-05
- train_batch_size: 32
- eval_batch_size: 4
- seed: 0
- distributed_type: multi-GPU
- num_devices: 64
- total_train_batch_size: 2048
- total_eval_batch_size: 256
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 10000
- training_steps: 187500

### Framework versions

- Transformers 4.25.1
- Pytorch 1.12.0a0+git664058f
- Datasets 2.7.1
- Tokenizers 0.12.1

## Acknowledgements

This model was trained for the paper "How beneficial is pre-training on a narrow domain-specific corpus for information extraction about photocatalytic water splitting?" by Taketomo Isazawa and Jacqueline M. Cole. J.M.C. is grateful for the BASF/Royal Academy of Engineering Research Chair in Data-Driven Molecular Engineering of Functional Materials, which includes PhD studentship support (for T.I.). This Chair is also partly supported by the Science and Technology Facilities Council. They are also indebted to the Argonne Leadership Computing Facility, which is a DOE Office of Science Facility, for use of its research resources, under contract No. DE-AC02-06CH11357.