LXMERT pre-trained model
Hello, congrats to all contributors for the awesome work with LXMERT! It is exciting to see multimodal transformers coming to hugginface/transformers. Of course, I immediately tried it out and to played with the demo.Question:Does the linelxmert_base = LxmertForPreTraining.from_pretrained("unc-nlp/lxmert-base-uncased")load an already pre-trained LXMERT model on the tasks enumerated in the original paper “(1) masked crossmodality language modeling, (2) masked object prediction via RoI-feature regression, (3) masked object prediction via detected-label classification, (4) cross-modality matching, and (5) image question answering.” (Tan & Bansal, 2019)?
2020-09-20T13:30:26Z
Tagging our LXMERT specialist@lysandre
2020-09-22T09:03:49Z
This question has been answered on Githubhereby@eltoto1219, the author of the huggingface implementation of LXMERT.
2020-09-22T12:57:02Z
Hello@lysandre, thanks for tagging the right person. Here is my github response with a new question:Hello@eltoto1219, thank you for the answer! I suppose it was a weird question from my part, since I was asking this to make sure that I am loading a pre-trained LXMERT model and not some random weights. Especially because I look at theoutput_lxmert['cross_relationship_score']of COCO images and captions (so not on some out of distribution images and captions), after I loaded LXMERT with the aforementioned codelxmert_base = LxmertForPreTraining.from_pretrained("unc-nlp/lxmert-base-uncased"). It seems thaton cross-modality matching LXMERT performs with 50% accuracy(random guessing). So I wanted to make sure that I load pre-trained weights on (4) cross-modality matching in the first place.New question: Do you know how it can be that LXMERT randomly guesses on cross-modality matching even it was pre-trained to deliver a score (after the Softmax, of course) of smaller 0.5 if the caption is not describing the image and a score bigger 0.5 it the caption and the image match?
2020-12-26T01:58:23Z
Any visual question answering demo? Thanks.
2020-12-29T06:54:06Z
Yes!Here it is
2021-12-25T16:55:45Z
Hi Suraj - I am looking for a good starting point to fine tune LXMERT for VQA task on a custom dataset. Could you please point me to something ?@valhalla