ai-forever
commited on
Commit
·
852065d
1
Parent(s):
213baf5
Update README.md
Browse files
README.md
CHANGED
@@ -15,10 +15,12 @@ Model was trained by [Sber AI](https://github.com/sberbank-ai) and [SberDevices]
|
|
15 |
|
16 |
# Model Description
|
17 |
|
18 |
-
RuDOLPH 350M is a fast and light text-image-text transformer (350M GPT-3) designed for a quick and easy fine-tuning setup for the solution of various tasks: from generating images by text description and image classification to visual question answering and more. This model demonstrates the power of Hyper-
|
|
|
|
|
19 |
|
20 |
# Sparse Attention Mask
|
21 |
|
22 |
-
The primary proposed method is to modify the sparse transformer's attention mask to better control multi-modalities. It allows us to calculate the transitions of modalities in both directions, unlike another similar work DALL-E Transformer, which used only one direction, "text to image". The proposed "image to right text" direction is achieved by extension sparse attention mask to the right for auto-repressively text generation with image condition without attention to left text.
|
23 |
|
24 |
<img src="https://raw.githubusercontent.com/shonenkov/ru-dolph/master/pics/attention_masks.png?token=AHV2MCP7BH3CQBAK74UVA7TB4CXQE" height="40" border="2"/>
|
|
|
15 |
|
16 |
# Model Description
|
17 |
|
18 |
+
RuDOLPH 350M is a fast and light text-image-text transformer (350M GPT-3) designed for a quick and easy fine-tuning setup for the solution of various tasks: from generating images by text description and image classification to visual question answering and more. This model demonstrates the power of Hyper-modality Transformers.
|
19 |
+
|
20 |
+
*(!!!) Hyper-modality means generalized multi-modal, e.g., model that consists of two multi-modal parts: text-2-image and image-2-text becomes text and image hyper-modality model*
|
21 |
|
22 |
# Sparse Attention Mask
|
23 |
|
24 |
+
The primary proposed method is to modify the sparse transformer's attention mask to better control multi-modalities and up to the next level with "hyper-modality". It allows us to calculate the transitions of modalities in both directions, unlike another similar work DALL-E Transformer, which used only one direction, "text to image". The proposed "image to right text" direction is achieved by extension sparse attention mask to the right for auto-repressively text generation with image condition without attention to left text.
|
25 |
|
26 |
<img src="https://raw.githubusercontent.com/shonenkov/ru-dolph/master/pics/attention_masks.png?token=AHV2MCP7BH3CQBAK74UVA7TB4CXQE" height="40" border="2"/>
|