Issue with KOSMOS-2 encoding and decoding
Hi@ydshieh,First of all I would like to say thank you for your work on Kosmos-2. I want to using Kosmos to for document AI. I am using trying to convert doclaynet dataset to embeddings. I read your code and research paper and found that first I need to convert bounding box to patch_XXXX thing. I tried that usingprocessorbut when I decode input_ids then I found miss match on the bounding box on the image.test = processor(images = [example['image']], text = [text], bboxes = [float_val])
test_decode = processor.decode(test['input_ids'][0])As you can see in the image that bounding boxes were overlapped and all.image1025×1025 253 KBCan you help me to figure this out?Here is the code:Google ColabThank you so much.
2024-01-24T13:59:47Z
Hi@Mit1208If I understand correctly, the attached contains the bounding boxes obtained via the processing(s) ofKosmos2Processorprocessed_text, entities = processor.post_process_generation(test_decode)And you want to demonstrate via this image that the bounding boxes are not matching the original input of bounding boxes (where you specified viaset_boxthen go throughnormalized_box(convert_box). Is this correct?I can see your image width/height is set to1025(I didn’t verify however).Kosmos2Processorsplits the image into a 32 x 32 grid. With the original input size224, each cell is of siez7x7. However, with your  image size1025, each cell will have size32x32which is quite large in document AI I believe.In order to verify this, you can compare the original bboxes input agains the final processed/computed output bboxes, and see if their differences are in the ragen of 32 (or even 64). If the differences are all inside this range, I wouldn’t say there is something wrong in the code ofKosmos2Processor. In this case, it’s just the limitation of such processing.If the differences are large than 32 or 64, something is likely to be wrong and I can take a more close look.If you don’t mind to train a model from scratch, there are some arguments could be changed to modify the default properties ofKosmos2Processor. I can share more information if you are donw to this.
2024-01-24T14:25:16Z
Thank you so much@ydshiehfor lookin into this issue. Yes your understanding is correct.You are right that the image size is 1025. About bounding boxes - labeled bounding box is ranging from 0-1025. I normalized it to 0-1 fromKosmos2Processor. I tried resizing image to 224 but found same results.You mentioned below, how can I do that?In order to verify this, you can compare the original bboxes input agains the final processed/computed output bboxes, and see if their differences are in the ragen of 32 (or even 64).I think training it from scratch is difficult because I don’t have GPUs. Do you think I can train the model on free Colab?
2024-01-24T14:48:32Z
It would be helpuful if you can:check first what I suggest aboveIn order to verify this, you can compare the original bboxes input agains the final processed/computed output bboxes, and see if their differences are in the ragen of 32 (or even 64). …ForI tried resizing image to 224 but found same results.Could you add the extra cells (from the end of the original notebook) showing thisHowever, I think with 224 document image, the word/line gaps are very smaller too. The ideal situation is that word/line gaps are large enough (compared to the word font size), so a bit off of the bounding bboxes won’t have undesired visual effect.
2024-01-24T14:52:27Z
I think training it from scratch is difficult because I don’t have GPUs. Do you think I can train the model on free Colab?First, this model is not trained on documents, I would doubt if it will work well (even with finetuning).train the model from scratch on free Colab → it probably could run but will be quite slow (data processing, training speed etc.)
2024-01-24T15:04:39Z
I have added 224*224 resizing code and output, it’s the same as the normal image. Can you see if my bbox conversion from (x, y, w, h) to [x, y, x+w, y+h] is correct and do I need to repeat that step after decoding fromKosmos2Processor?It seems I am very close but missing something small.I just wanted to see how this model reacts to the document compared to LayoutLMv3.
2024-01-24T17:04:55Z
Hi , thanks. I see you post the desired outputs. Looking both visually, I think it’s really just the limitation ofKosmos2Processor(which using cells in a 32x32 grid), which works well in a common images with smaller size. For documents, the cell approaches isn’t good as we need higher precision of the locations.
2024-01-24T17:32:22Z
Hi, is there a workaround for this like, creating more patches to handle this? I don’t have a lot of experience other than fine-tuning models but I can give it a shot. Can you refer to some documentation or tutorial which I can follow to learn and overcome this issue?Thanks.
2024-01-24T18:09:21Z
Yes, you can specifynum_patch_index_tokenswhen creatingKosmos2Processor. (I think it works when we usefrom_pretrainedand passingnum_patch_index_tokens)And while doing process(es), we need to passnum_patches_per_side.(seeclean_text_and_extract_entities_with_bboxes)(post_process_generationdoesn’t have this argument, which I should probably update this method)
2024-01-24T18:09:33Z
Let me know how it goes!
2024-01-26T12:21:32Z
Thanks, sure I will keep you posted.
2024-01-28T19:01:00Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.