> # Cloned from https://github.com/amazon-science/mm-cot

# Multimodal Chain-of-Thought Reasoning in Language Models

<h5 align="center"><i>"Imagine learning a textbook without figures or tables."</i></h5>

Multimodal-CoT incorporates vision features in a decoupled training framework. The framework consists of two training stages: (i) rationale generation and (ii) answer inference. Both stages share the same model architecture but differ in the input and output.

![](vision_features/mm-cot.png)


## Requirements

Install all required python dependencies:

```
pip install -r requirements.txt
```

## Datasets

Download the dataset from the following repository:

```
https://github.com/lupantech/ScienceQA/tree/main/data
```

Download the extracted vision features from [vision_features](https://drive.google.com/file/d/13B0hc_F_45-UlqPLKSgRz-ALtFQ8kIJr/view?usp=share_link) and unzip the files under `vision_features`

## Instructions

### Training 

```
# rationale generation
CUDA_VISIBLE_DEVICES=0,1 python main.py \
    --model allenai/unifiedqa-t5-base \
    --user_msg rationale --img_type detr \
    --bs 8 --eval_bs 4 --eval_acc 10 --output_len 512 \
    --final_eval --prompt_format QCM-LE

# answer inference
CUDA_VISIBLE_DEVICES=0,1 python main.py \
    --model allenai/unifiedqa-t5-base \
    --user_msg answer --img_type detr \
    --bs 8 --eval_bs 4 --eval_acc 10 --output_len 64 \
    --final_eval --prompt_format QCMG-A \
    --eval_le experiments/rationale_allenai-unifiedqa-t5-base_detr_QCM-LE_lr5e-05_bs16_op512_ep20/predictions_ans_eval.json \
    --test_le experiments/rationale_allenai-unifiedqa-t5-base_detr_QCM-LE_lr5e-05_bs16_op512_ep20/predictions_ans_test.json
```

### Inference 

Our trained models are available at [models](https://drive.google.com/file/d/1FtTYOJPHnWnFfCxNC6M3gar4RAX5E21b/view?usp=share_link). To use our trained models, please put the them under the ```models``` folder.

```
# rationale generation
CUDA_VISIBLE_DEVICES=0,1 python main.py \
    --model allenai/unifiedqa-t5-base \
    --user_msg rationale --img_type detr \
    --bs 8 --eval_bs 4 --eval_acc 10 --output_len 512 \
    --final_eval --prompt_format QCM-LE \
    --evaluate_dir models/MM-CoT-UnifiedQA-base-Rationale

# answer inference
CUDA_VISIBLE_DEVICES=0,1 python main.py \
    --model allenai/unifiedqa-t5-base \
    --user_msg answer --img_type detr \
    --bs 8 --eval_bs 4 --eval_acc 10 --output_len 64 \
    --final_eval --prompt_format QCMG-A \
    --eval_le models/rationale/predictions_ans_eval.json \
    --test_le models/rationale/predictions_ans_test.json \
    --evaluate_dir models/MM-CoT-UnifiedQA-base-Answer
```

## Citing MM-CoT

```
@article{zhang2023multicot,
  title={Multimodal Chain-of-Thought Reasoning in Language Models},
  author={Zhang, Zhuosheng and Zhang, Aston and Li, Mu and Zhao, Hai and Karypis, George and Smola, Alex},
  journal={arXiv preprint arXiv:2302.00923},
  year={2023}
}
```

## License

This project is licensed under the Apache-2.0 License.

## Acknowledgement

Part of our codes are adapted from [ScienceQA](https://github.com/lupantech/ScienceQA) and [Transformers](https://github.com/huggingface/transformers).

We thank Pan Lu for providing parameter size for ScienceQA baselines.

{}
region:us
TAGS
#region-us 

> # Cloned from URL

# Multimodal Chain-of-Thought Reasoning in Language Models

<h5 align="center"><i>"Imagine learning a textbook without figures or tables."</i></h5>

Multimodal-CoT incorporates vision features in a decoupled training framework. The framework consists of two training stages: (i) rationale generation and (ii) answer inference. Both stages share the same model architecture but differ in the input and output.

![](vision_features/URL)


## Requirements

Install all required python dependencies:



## Datasets

Download the dataset from the following repository:



Download the extracted vision features from vision_features and unzip the files under 'vision_features'

## Instructions

### Training 



### Inference 

Our trained models are available at models. To use our trained models, please put the them under the  folder.



## Citing MM-CoT



## License

This project is licensed under the Apache-2.0 License.

## Acknowledgement

Part of our codes are adapted from ScienceQA and Transformers.

We thank Pan Lu for providing parameter size for ScienceQA baselines.

# Cloned from URL # Multimodal Chain-of-Thought Reasoning in Language Models

<h5 align="center"><i>"Imagine learning a textbook without figures or tables."</i></h5>

Multimodal-CoT incorporates vision features in a decoupled training framework. The framework consists of two training stages: (i) rationale generation and (ii) answer inference. Both stages share the same model architecture but differ in the input and output.

![](vision_features/URL) ## Requirements

Install all required python dependencies: ## Datasets

Download the dataset from the following repository:



Download the extracted vision features from vision_features and unzip the files under 'vision_features' ## Instructions ### Training ### Inference 

Our trained models are available at models. To use our trained models, please put the them under the  folder. ## Citing MM-CoT ## License

This project is licensed under the Apache-2.0 License. ## Acknowledgement

Part of our codes are adapted from ScienceQA and Transformers.

We thank Pan Lu for providing parameter size for ScienceQA baselines.
TAGS
#region-us 
 # Cloned from URL # Multimodal Chain-of-Thought Reasoning in Language Models

<h5 align="center"><i>"Imagine learning a textbook without figures or tables."</i></h5>

Multimodal-CoT incorporates vision features in a decoupled training framework. The framework consists of two training stages: (i) rationale generation and (ii) answer inference. Both stages share the same model architecture but differ in the input and output.

![](vision_features/URL) ## Requirements

Install all required python dependencies: ## Datasets

Download the dataset from the following repository:



Download the extracted vision features from vision_features and unzip the files under 'vision_features' ## Instructions ### Training ### Inference 

Our trained models are available at models. To use our trained models, please put the them under the  folder. ## Citing MM-CoT ## License

This project is licensed under the Apache-2.0 License. ## Acknowledgement

Part of our codes are adapted from ScienceQA and Transformers.

We thank Pan Lu for providing parameter size for ScienceQA baselines.
TAGS
#region-us 
# Cloned from URL# Multimodal Chain-of-Thought Reasoning in Language Models

<h5 align="center"><i>"Imagine learning a textbook without figures or tables."</i></h5>

Multimodal-CoT incorporates vision features in a decoupled training framework. The framework consists of two training stages: (i) rationale generation and (ii) answer inference. Both stages share the same model architecture but differ in the input and output.

![](vision_features/URL)## Requirements

Install all required python dependencies:## Datasets

Download the dataset from the following repository:



Download the extracted vision features from vision_features and unzip the files under 'vision_features'## Instructions### Training### Inference 

Our trained models are available at models. To use our trained models, please put the them under the  folder.## Citing MM-CoT## License

This project is licensed under the Apache-2.0 License.## Acknowledgement

Part of our codes are adapted from ScienceQA and Transformers.

We thank Pan Lu for providing parameter size for ScienceQA baselines.