region:us
## AI Journey 2023 Baseline solution

This solution is inspired by the methodologies of [FROMAGe](https://github.com/kohjingyu/fromage) and [Kosmos-1](https://github.com/microsoft/unilm/tree/master/kosmos-2). It primarily employs these approaches to fine-tune the linear mapping from visual and audio vector spaces into the language model-decoder's vector space. Subsequently, the response is generated exclusively using the intact language model. 

As a modality encoder, we utilize [ImageBind](https://github.com/facebookresearch/ImageBind). This encoder has been trained specifically for understanding images, audio, and text and other data formats in a shared embedding space.

During the training phase, the weights of the encoder and the language model remain frozen. The exceptions to this are the additional embeddings for two tokens marking the beginning and end of the respective modalities in the language model: `<SOI>`, `<EOI>` and `<SOA>`, `<EOA>` (S, E — Start, End; I,A — Image, Audio).

Training is leveraged using four datasets: [VisualDialogues](https://visualdialog.org/data), [COCO Captions](https://cocodataset.org/#home), [Clotho v. 2.1](https://zenodo.org/record/3490684) and [Clotho-AQA](https://zenodo.org/record/6473207). The core training objective is next token prediction with CrossEntropy loss. The general architecture is  illustrated here:

<p align="center">
<img alt="Baseline model architecture" src="./baseline.png" width="100%">
</p>

## Training
To reproduce training, please run [notebook](./baseline.ipynb) after installing requirements:
```
pip install requirements.txt
```