# HoogBERTa

This repository includes the Thai pretrained language representation (HoogBERTa_base) and can be used for **Feature Extraction and Masked Language Modeling Tasks**.


# Documentation


## Prerequisite
Since we use subword-nmt BPE encoding, input needs to be pre-tokenize using [BEST](https://huggingface.co/datasets/best2009) standard before inputting into HoogBERTa 
```
pip install attacut
```

## Getting Start
To initialize the model from hub, use the following commands
```python
from transformers import AutoTokenizer, AutoModel
from attacut import tokenize
import torch

tokenizer = AutoTokenizer.from_pretrained("lst-nectec/HoogBERTa")
model = AutoModel.from_pretrained("lst-nectec/HoogBERTa")
```

To extract token features, based on the RoBERTa architecture, use the following commands

```python
model.eval()
sentence = "วันที่ 12 มีนาคมนี้ ฉันจะไปเที่ยววัดพระแก้ว ที่กรุงเทพ"
all_sent = []
sentences = sentence.split(" ")
for sent in sentences:
    all_sent.append(" ".join(tokenize(sent)).replace("_","[!und:]"))

sentence = " _ ".join(all_sent)
tokenized_text = tokenizer(sentence, return_tensors = 'pt')
token_ids = tokenized_text['input_ids']

with torch.no_grad():
  features = model(**tokenized_text, output_hidden_states = True).hidden_states[-1]
```

For batch processing,

```python
model.eval()
sentenceL = ["วันที่ 12 มีนาคมนี้","ฉันจะไปเที่ยววัดพระแก้ว ที่กรุงเทพ"]
inputList = []
for sentX in sentenceL:
  sentences = sentX.split(" ")
  all_sent = []
  for sent in sentences:
      all_sent.append(" ".join(tokenize(sent)).replace("_","[!und:]"))

  sentence = " _ ".join(all_sent)
  inputList.append(sentence)
tokenized_text = tokenizer(inputList, padding = True, return_tensors = 'pt')
token_ids = tokenized_text['input_ids']

with torch.no_grad():
    features = model(**tokenized_text, output_hidden_states = True).hidden_states[-1]
```

To use HoogBERTa as an embedding layer, use

```python
with torch.no_grad():
  features = model(token_ids, output_hidden_states = True).hidden_states[-1] # where token_ids is a tensor with type "long".
```

# Huggingface Models
1. `HoogBERTaEncoder`
 - [HoogBERTa](https://huggingface.co/lst-nectec/HoogBERTa): `Feature Extraction` and `Mask Language Modeling`
2. `HoogBERTaMuliTaskTagger`:
 - [HoogBERTa-NER-lst20](https://huggingface.co/lst-nectec/HoogBERTa-NER-lst20): `Named-entity recognition (NER)` based on LST20
 - [HoogBERTa-POS-lst20](https://huggingface.co/lst-nectec/HoogBERTa-POS-lst20): `Part-of-speech tagging (POS)` based on LST20
 - [HoogBERTa-SENTENCE-lst20](https://huggingface.co/lst-nectec/HoogBERTa-SENTENCE-lst20): `Clause Boundary Classification` based on LST20


# Citation

Please cite as:

``` bibtex
@inproceedings{porkaew2021hoogberta,
  title = {HoogBERTa: Multi-task Sequence Labeling using Thai Pretrained Language Representation},
  author = {Peerachet Porkaew, Prachya Boonkwan and Thepchai Supnithi},
  booktitle = {The Joint International Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP 2021)},
  year = {2021},
  address={Online}
}
```

Download full-text [PDF](https://drive.google.com/file/d/1hwdyIssR5U_knhPE2HJigrc0rlkqWeLF/view?usp=sharing)  
Check out the code on [Github](https://github.com/lstnlp/HoogBERTa)
{"language": ["th"], "license": "mit", "library_name": "transformers", "datasets": ["best2009", "scb_mt_enth_2020", "oscar", "wikipedia"], "widget": [{"text": "\u0e27\u0e31\u0e19 \u0e17\u0e35\u0e48 _ 12 _ \u0e21\u0e35\u0e19\u0e32\u0e04\u0e21 \u0e19\u0e35\u0e49 _ \u0e09\u0e31\u0e19 \u0e08\u0e30 \u0e44\u0e1b <mask> \u0e27\u0e31\u0e14\u0e1e\u0e23\u0e30\u0e41\u0e01\u0e49\u0e27 _ \u0e17\u0e35\u0e48 \u0e01\u0e23\u0e38\u0e07\u0e40\u0e17\u0e1e"}]}
transformers pytorch safetensors roberta fill-mask th dataset:best2009 dataset:scb_mt_enth_2020 dataset:oscar dataset:wikipedia license:mit autotrain_compatible endpoints_compatible region:us
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #th #dataset-best2009 #dataset-scb_mt_enth_2020 #dataset-oscar #dataset-wikipedia #license-mit #autotrain_compatible #endpoints_compatible #region-us 

# HoogBERTa

This repository includes the Thai pretrained language representation (HoogBERTa_base) and can be used for Feature Extraction and Masked Language Modeling Tasks.


# Documentation


## Prerequisite
Since we use subword-nmt BPE encoding, input needs to be pre-tokenize using BEST standard before inputting into HoogBERTa 


## Getting Start
To initialize the model from hub, use the following commands


To extract token features, based on the RoBERTa architecture, use the following commands



For batch processing,



To use HoogBERTa as an embedding layer, use



# Huggingface Models
1. 'HoogBERTaEncoder'
 - HoogBERTa: 'Feature Extraction' and 'Mask Language Modeling'
2. 'HoogBERTaMuliTaskTagger':
 - HoogBERTa-NER-lst20: 'Named-entity recognition (NER)' based on LST20
 - HoogBERTa-POS-lst20: 'Part-of-speech tagging (POS)' based on LST20
 - HoogBERTa-SENTENCE-lst20: 'Clause Boundary Classification' based on LST20


Please cite as:



Download full-text PDF  
Check out the code on Github
# HoogBERTa

This repository includes the Thai pretrained language representation (HoogBERTa_base) and can be used for Feature Extraction and Masked Language Modeling Tasks. # Documentation ## Prerequisite
Since we use subword-nmt BPE encoding, input needs to be pre-tokenize using BEST standard before inputting into HoogBERTa ## Getting Start
To initialize the model from hub, use the following commands


To extract token features, based on the RoBERTa architecture, use the following commands



For batch processing,



To use HoogBERTa as an embedding layer, use # Huggingface Models
1. 'HoogBERTaEncoder'
 - HoogBERTa: 'Feature Extraction' and 'Mask Language Modeling'
2. 'HoogBERTaMuliTaskTagger':
 - HoogBERTa-NER-lst20: 'Named-entity recognition (NER)' based on LST20
 - HoogBERTa-POS-lst20: 'Part-of-speech tagging (POS)' based on LST20
 - HoogBERTa-SENTENCE-lst20: 'Clause Boundary Classification' based on LST20


Please cite as:



Download full-text PDF  
Check out the code on Github
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #th #dataset-best2009 #dataset-scb_mt_enth_2020 #dataset-oscar #dataset-wikipedia #license-mit #autotrain_compatible #endpoints_compatible #region-us 
 # HoogBERTa

This repository includes the Thai pretrained language representation (HoogBERTa_base) and can be used for Feature Extraction and Masked Language Modeling Tasks. # Documentation ## Prerequisite
Since we use subword-nmt BPE encoding, input needs to be pre-tokenize using BEST standard before inputting into HoogBERTa ## Getting Start
To initialize the model from hub, use the following commands


To extract token features, based on the RoBERTa architecture, use the following commands



For batch processing,



To use HoogBERTa as an embedding layer, use # Huggingface Models
1. 'HoogBERTaEncoder'
 - HoogBERTa: 'Feature Extraction' and 'Mask Language Modeling'
2. 'HoogBERTaMuliTaskTagger':
 - HoogBERTa-NER-lst20: 'Named-entity recognition (NER)' based on LST20
 - HoogBERTa-POS-lst20: 'Part-of-speech tagging (POS)' based on LST20
 - HoogBERTa-SENTENCE-lst20: 'Clause Boundary Classification' based on LST20


Please cite as:



Download full-text PDF  
Check out the code on Github
TAGS
#transformers #pytorch #safetensors #roberta #fill-mask #th #dataset-best2009 #dataset-scb_mt_enth_2020 #dataset-oscar #dataset-wikipedia #license-mit #autotrain_compatible #endpoints_compatible #region-us 
# HoogBERTa

This repository includes the Thai pretrained language representation (HoogBERTa_base) and can be used for Feature Extraction and Masked Language Modeling Tasks.# Documentation## Prerequisite
Since we use subword-nmt BPE encoding, input needs to be pre-tokenize using BEST standard before inputting into HoogBERTa## Getting Start
To initialize the model from hub, use the following commands


To extract token features, based on the RoBERTa architecture, use the following commands



For batch processing,



To use HoogBERTa as an embedding layer, use# Huggingface Models
1. 'HoogBERTaEncoder'
 - HoogBERTa: 'Feature Extraction' and 'Mask Language Modeling'
2. 'HoogBERTaMuliTaskTagger':
 - HoogBERTa-NER-lst20: 'Named-entity recognition (NER)' based on LST20
 - HoogBERTa-POS-lst20: 'Part-of-speech tagging (POS)' based on LST20
 - HoogBERTa-SENTENCE-lst20: 'Clause Boundary Classification' based on LST20


Please cite as:



Download full-text PDF  
Check out the code on Github