|
--- |
|
language: en |
|
tags: |
|
- bert |
|
- business |
|
- finance |
|
license: cc-by-4.0 |
|
datasets: |
|
- CompanyWeb |
|
- MD&A |
|
- S2ORC |
|
--- |
|
|
|
# BusinessBERT |
|
|
|
An industry-sensitive language model for business applications pretrained on business communication corpora. The model incorporates industry classification (IC) as a pretraining objective besides masked language modeling (MLM). |
|
|
|
It was introduced in |
|
[this paper]() and released in |
|
[this repository](). |
|
|
|
## Model description |
|
|
|
We introduce BusinessBERT, an industry-sensitive language model for business applications. The advantage of the model is the training approach focused on incorporating industry information relevant for business related natural language processing (NLP) tasks. |
|
We compile three large-scale textual corpora consisting of annual disclosures, company website content and scientific literature representing business communication. In total, the corpora include 2.23 billion token. |
|
BusinessBERT builds upon the bidirectional encoder representations from transformer architecture (BERT) and embeds industry information during pretraining in two ways: (1) The business communication corpora contain a variety of industry-specific terminology; (2) We employ industry classification (IC) as an additional pretraining objective for text documents originating from companies. |
|
|
|
## Intended uses & limitations |
|
|
|
The model is intended to be fine-tuned on business related NLP tasks, i.e. sequence classification, named entity recognition, sentiment analysis or question answering. |
|
|
|
### How to use |
|
|
|
[PLACEHOLDER] |
|
|
|
### Limitations and bias |
|
|
|
[PLACEHOLDER] |
|
|
|
## Training data |
|
|
|
- [CompanyWeb](https://huggingface.co/datasets/anonymousparrot01/CompanyWeb): 0.77 billion token, 3.5 GB raw text file |
|
- [MD&A Disclosures](https://data.caltech.edu/records/1249): 1.06 billion token, 5.1 GB raw text file |
|
- [Semantic Scholar Open Research Corpus](https://api.semanticscholar.org/corpus): 0.40 billion token, 1.9 GB raw text file |
|
|
|
## Evaluation results |
|
|
|
[PLACEHOLDER] |
|
|
|
<!-- When fine-tuned on downstream tasks, this model achieves the following results: |
|
|
|
Glue test results: |
|
|
|
| Task | MNLI-(m/mm) | QQP | QNLI | SST-2 | CoLA | STS-B | MRPC | RTE | Average | |
|
|:----:|:-----------:|:----:|:----:|:-----:|:----:|:-----:|:----:|:----:|:-------:| |
|
| | 84.6/83.4 | 71.2 | 90.5 | 93.5 | 52.1 | 85.8 | 88.9 | 66.4 | 79.6 | --> |
|
|
|
|
|
### BibTeX entry and citation info |
|
|
|
```bibtex |
|
@misc{title_year, |
|
title={TITLE}, |
|
author={AUTHORS}, |
|
year={YEAR}, |
|
} |
|
``` |
