Finetuning Segment Anything and automatic prediction
I am trying to fine-tune the Segment Anything (SAM) model following thedemo notebook(credits:@nielsrand@ybelkada).In my case, I have trained on a custom dataset. Now, I want it to predict masks automatically (without any prompts like bounding boxes). How to do it?I tried a few things (input_boxes=None,input_boxes = [0,0,img_width, img_height], usingpipeline("mask-generator", model=my_model, processor = my_processor)fromhereand loading my checkpoint like in the officialautomatic_mask_generator_example.ipynb) but they didn’t work.
2023-06-05T12:52:17Z
Hi,SAM always expects a prompt, which could be a bounding box, mask, text or point (the authors didn’t release the text prompt capability). SAM just generates a mask given an image + a prompt.Theautomatic_mask_generatorworks by generating a lot of point prompts at fixed positions in the image, which enable SAM to generate masks given those prompts. Those are then postprocessed using NMS to remove duplicate detections:Specifically, we prompt SAM with a 16×16 regular grid of foreground points resulting in 768 predicted masks (3 per point). Redundant masks are removed by NMS.So in any case, SAM requires prompts. You could for instance first run a zero-shot object detector on your image to get bounding boxes, and then run SAM to get masks given those bounding boxes. That’s what theGrounding DINO-SAM repodoes
2023-06-07T09:42:13Z
Thanks for the reply! SAM does require prompts, but I was hoping for some pipeline that does the sampling internally.