[Owlv2 - image_guided_detection - embed_image_query] Why choosing the least similar box from selected ones?
I’m trying to understand the owlv2 image_guided_detection and have a question.From this tutorial about OWLv2zero_oneshot_owlv2_ObjectionDetection, the author said that the image_guided_detection part uses a heuristic way toget the patch in the source image which most likely contains an objectLooking at the source code athttps://github.com/huggingface/transformers/blob/main/src/transformers/models/owlv2/modeling_owlv2.pyThe heuristic he mentioned I believe is here:iou_threshold = torch.max(ious) * 0.8

            selected_inds = (ious[0] >= iou_threshold).nonzero()
            if selected_inds.numel():
                selected_embeddings = class_embeds[i][selected_inds.squeeze(1)]
                mean_embeds = torch.mean(class_embeds[i], axis=0)
                mean_sim = torch.einsum("d,id->i", mean_embeds, selected_embeddings)
                best_box_ind = selected_inds[torch.argmin(mean_sim)]
                best_class_embeds.append(class_embeds[i][best_box_ind])
                best_box_indices.append(best_box_ind)So what I understand from this code:Select a list of bboxCalculate the mean of embedding of these bboxCalculate the similarity of the mean_embedding and all bbox_embeddingsSelect the bbox which is the least similar to the mean viabest_box_ind = selected_inds[torch.argmin(mean_sim)]So, why choose the least similar here instead of the most similar one with argmax? We want to choose a box closest to the mean, right?Thanks
2023-11-24T10:20:39Z
[Update]Maybe the reason for choosing the least similar is to remove noise because when I change from argmin to argmax. I have a lot of False Positives ( even when the chosen bounding box is not different too much for both cases, very weird)image726×664 52 KBStill not sure what is the best way to work with OwlV2 for image-guided detection, anyone know the best practices?Thanks
2023-11-24T13:43:12Z
The reason can be found in the original implementation of OWLv2 from scenic:github.comgoogle-research/scenic/blob/main/scenic/projects/owl_vit/notebooks/inference.py"""Code for running (interactive) inference with OWL-ViT models."""

import dataclasses
import functools
from typing import Any, Dict, Tuple

from flax import linen as nn
import jax
import jax.numpy as jnp
import ml_collections
import numpy as np
from scenic.model_lib.base_models import box_utils
from scenic.projects.owl_vit.notebooks import numpy_cache
from scipy import special as sp_special
from skimage import transform as skimage_transform
import tensorflow as tf

sigmoid = sp_special.expit  # Sigmoid is a more familiar name.
QUERY_PAD_BIN_SIZE = 50This file has been truncated.show original# Due to the DETR style bipartite matching loss, only one embedding
# feature for each object is "good" and the rest are "background." To find
# the one "good" feature we use the heuristic that it should be dissimilar
# to the mean embedding.Does it also mean that OWLv2 image-guided-detection is very sensible to noise? just a very small difference in the query bounding box and the result is completely wrong