
# BigBird base (ITC) fine-tuned on MP-DocVQA

This is BigBird-base trained on TriviaQA from [Google hub](https://huggingface.co/google/bigbird-base-trivia-itc) and fine-tuned on Multipage DocVQA (MP-DocVQA) dataset.
* Due to Huggingface implementation, the global tokens are defined according to the Internal Transformer Construction (ITC) strategy.

This model was used as a baseline in [Hierarchical multimodal transformers for Multi-Page DocVQA](https://arxiv.org/pdf/2212.05935.pdf).
- Results on the MP-DocVQA dataset are reported in Table 2.
- Training hyperparameters can be found in Table 8 of Appendix D.

## How to use

How to use this model to perform inference on a sample question and context in PyTorch:

```python
from transformers import BigBirdForQuestionAnswering, BigBirdTokenizerFast

# by default its in `block_sparse` mode with num_random_blocks=3, block_size=64
model = BigBirdForQuestionAnswering.from_pretrained("rubentito/bigbird-base-itc-mpdocvqa")

# you can change `attention_type` to full attention like this:
model = BigBirdForQuestionAnswering.from_pretrained("rubentito/bigbird-base-itc-mpdocvqa", attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = BigBirdForQuestionAnswering.from_pretrained("rubentito/bigbird-base-itc-mpdocvqa", block_size=16, num_random_blocks=2)

tokenizer = BigBirdTokenizerFast.from_pretrained("rubentito/bigbird-base-itc-mpdocvqa")

question = "Replace me by any text you'd like."
context = "Put some context for answering"

encoded_input = tokenizer(question, context, return_tensors='pt')
output = model(**encoded_input)

start_pos = torch.argmax(output.start_logits, dim=-1).item()
end_pos = torch.argmax(output.end_logits, dim=-1).item()

context_tokens = tokenizer.convert_ids_to_tokens(encoded_input["input_ids"][0].tolist())
answer_tokens = context_tokens[start_pos: end_pos]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))
```

## Metrics
**Average Normalized Levenshtein Similarity (ANLS)**

The standard metric for text-based VQA tasks (ST-VQA and DocVQA). It evaluates the method's reasoning capabilities while smoothly penalizes OCR recognition errors.
Check [Scene Text Visual Question Answering](https://arxiv.org/abs/1905.13648) for detailed information.

**Answer Page Prediction Accuracy (APPA)**

In the MP-DocVQA task, the models can provide the index of the page where the information required to answer the question is located. For this subtask accuracy is used to evaluate the predictions: i.e. if the predicted page is correct or not.
Check [Hierarchical multimodal transformers for Multi-Page DocVQA](https://arxiv.org/abs/2212.05935) for detailed information.

## Model results

Extended experimentation can be found in Table 2 of [Hierarchical multimodal transformers for Multi-Page DocVQA](https://arxiv.org/pdf/2212.05935.pdf).
You can also check the live leaderboard at the [RRC Portal](https://rrc.cvc.uab.es/?ch=17&com=evaluation&task=4).
| Model 		 																	| HF name								| Parameters 	|	ANLS 		| APPA		|
|-----------------------------------------------------------------------------------|:--------------------------------------|:-------------:|:-------------:|:---------:|
| [Bert large](https://huggingface.co/rubentito/bert-large-mpdocvqa)	            | rubentito/bert-large-mpdocvqa			| 334M 			| 0.4183 		| 51.6177 	|
| [Longformer base](https://huggingface.co/rubentito/longformer-base-mpdocvqa)		| rubentito/longformer-base-mpdocvqa	| 148M			| 0.5287		| 71.1696 	|
| [**BigBird ITC base**](https://huggingface.co/rubentito/bigbird-base-itc-mpdocvqa)| rubentito/bigbird-base-itc-mpdocvqa	| 131M			| 0.4929		| 67.5433 	|
| [LayoutLMv3 base](https://huggingface.co/rubentito/layoutlmv3-base-mpdocvqa)		| rubentito/layoutlmv3-base-mpdocvqa	| 125M 			| 0.4538		| 51.9426 	|
| [T5 base](https://huggingface.co/rubentito/t5-base-mpdocvqa)						| rubentito/t5-base-mpdocvqa			| 223M 			| 0.5050		| 0.0000 	|
| [Hi-VT5](https://huggingface.co/rubentito/hivt5-base-mpdocvqa)  					| rubentito/hivt5-base-mpdocvqa 		| 316M 			| 0.6201		| 79.23		|

## Citation Information 

```tex
@article{tito2022hierarchical,
  title={Hierarchical multimodal transformers for Multi-Page DocVQA},
  author={Tito, Rub{\`e}n and Karatzas, Dimosthenis and Valveny, Ernest},
  journal={arXiv preprint arXiv:2212.05935},
  year={2022}
}
```
{"language": ["en"], "license": "gpl-3.0", "tags": ["DocVQA", "Document Question Answering", "Document Visual Question Answering"], "datasets": ["rubentito/mp-docvqa"], "base_model": "google/bigbird-base-trivia-itc"}
transformers pytorch big_bird question-answering DocVQA Document Question Answering Document Visual Question Answering en dataset:rubentito/mp-docvqa arxiv:2212.05935 arxiv:1905.13648 base_model:google/bigbird-base-trivia-itc license:gpl-3.0 endpoints_compatible region:us
TAGS
#transformers #pytorch #big_bird #question-answering #DocVQA #Document Question Answering #Document Visual Question Answering #en #dataset-rubentito/mp-docvqa #arxiv-2212.05935 #arxiv-1905.13648 #base_model-google/bigbird-base-trivia-itc #license-gpl-3.0 #endpoints_compatible #region-us 

BigBird base (ITC) fine-tuned on MP-DocVQA
==========================================


This is BigBird-base trained on TriviaQA from Google hub and fine-tuned on Multipage DocVQA (MP-DocVQA) dataset.


* Due to Huggingface implementation, the global tokens are defined according to the Internal Transformer Construction (ITC) strategy.


This model was used as a baseline in Hierarchical multimodal transformers for Multi-Page DocVQA.


* Results on the MP-DocVQA dataset are reported in Table 2.
* Training hyperparameters can be found in Table 8 of Appendix D.


How to use
----------


How to use this model to perform inference on a sample question and context in PyTorch:


Metrics
-------


Average Normalized Levenshtein Similarity (ANLS)


The standard metric for text-based VQA tasks (ST-VQA and DocVQA). It evaluates the method's reasoning capabilities while smoothly penalizes OCR recognition errors.
Check Scene Text Visual Question Answering for detailed information.


Answer Page Prediction Accuracy (APPA)


In the MP-DocVQA task, the models can provide the index of the page where the information required to answer the question is located. For this subtask accuracy is used to evaluate the predictions: i.e. if the predicted page is correct or not.
Check Hierarchical multimodal transformers for Multi-Page DocVQA for detailed information.


Model results
-------------


Extended experimentation can be found in Table 2 of Hierarchical multimodal transformers for Multi-Page DocVQA.
You can also check the live leaderboard at the RRC Portal.




TAGS
#transformers #pytorch #big_bird #question-answering #DocVQA #Document Question Answering #Document Visual Question Answering #en #dataset-rubentito/mp-docvqa #arxiv-2212.05935 #arxiv-1905.13648 #base_model-google/bigbird-base-trivia-itc #license-gpl-3.0 #endpoints_compatible #region-us 

TAGS
#transformers #pytorch #big_bird #question-answering #DocVQA #Document Question Answering #Document Visual Question Answering #en #dataset-rubentito/mp-docvqa #arxiv-2212.05935 #arxiv-1905.13648 #base_model-google/bigbird-base-trivia-itc #license-gpl-3.0 #endpoints_compatible #region-us 
