Visual Tokenization / Masking In BEIT & LayoutLMv3
tldr questions:Why do papers likeLayoutLMv3mention visual tokenizers that are absent from all open-source implementations?Why do most BEIT implementations lack the visual tokenizer?Why do pixel values go through PatchEmbedding straight into a decoder without any tokenization?What is the relationship between Patch Embed, self.mask_token = nn.Parameter() and visual tokenizers.Background:I’m coding the heads forLayoutLMv3pretraining. The paper claims to use the visual tokenizer fromDITto tokenize image patches. DIT is essentially BEIT with a dVAE trained on documents.But, when I look at the code in HuggingFace and TIMM for BEIT, there’s no tokenization of image patches.timm: BEIThugging-face: BEITThe pixel values get transformed into embeds that go straight into the encoder.InMicrosofts BEIT implementationI managed to find an example that makes use of visual tokenizers.After successfully using the visual tokenizer, my confusion only increased and led me toquestion number 4.Thank you for reading and for your attention. Any light shed on this subject would be amazing.
2022-12-23T13:03:12Z
moment.Is theself.mask_token = nn.Parameter(torch.zeros(1, 1, embed_dim))what we replace the visual embeddings to be masked with? Is the visual token a learnable parameter?