MMBT Model (Resnet and BERT) for multimodal embeddings
Hi! I’m trying to use the librarys implementation ofMultimodal Bitransformers(Kiela et all.) on how to classify images and text simultaneously.I’ve found it hard, because there is very little documentation, and no examples.In particular i’ve been having a hard time figuring out how to pass the encoded image with the tokenized text to the already intialized model.If anyone has already worked with this implementation i could really use some helpThanks
2020-12-24T16:14:33Z
Hey, check out thisexample.
2021-10-21T07:31:55Z
Any luck with this one? What were the results?
2021-11-10T09:40:01Z
I tried to reproduce the code of the example and found a couple of issues:in the inputs ofMMBTForClassificationmodel there should be also a keyreturn_dictotherwise it will not workdepending on how you one-hot encode the labels you might want to cast the vectors as float for the loss computationI also have a couple of questions:the data loader in the example returns batches containingimage_start_tokenandimage_end_tokenthat have been defined as the first and last special tokens of the tokenized sentence. I’m somehow puzzled with this and I’ll try to investigate further why…do you know why a gradient accumulation step phase is used? is that normal that after that there is amodel.zero_grad()?the criterion chosen for the example isnn.BCEWithLogitsLoss(pos_weight=label_weights)and I was wondering why exactly. Is that fine for multiclass classification? What would be the equivalent of keras categorical entropy or sparse categorical entropy?