Try to fine tune SegFormer for binary semantic segmentation but metrics are nan
Hi, I’m trying to fine tune SegFormer for binary semantic segmentation. I looked over some quick tutorials likesemantic segmentationandfine-tune-segformerbut I could not run my case.I have two classes that are car and background.id2label = { 0: 'background', 1: 'car' }
label2id = { 'background':0, 'car':1}
num_labels = len(id2label)First, I got thesize mismatcherror when I try giving input id2label and label2id to the model variable. Then I convert the ignore_mismatched_sizes=True (I do not know that it is a proper way)model = SegformerForSemanticSegmentation.from_pretrained("nvidia/segformer-b0-finetuned-ade-512-512", id2label=id2label, label2id=label2id, ignore_mismatched_sizes=True)Then when I try to train the model with mean_iou metric from evaluate library, I got nan values for all metrics with these warnings:/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:258: RuntimeWarning: invalid value encountered in double_scalars
  all_acc = total_area_intersect.sum() / total_area_label.sum()
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:259: RuntimeWarning: invalid value encountered in divide
  iou = total_area_intersect / total_area_union
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:260: RuntimeWarning: invalid value encountered in divide
  acc = total_area_intersect / total_area_label
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:262: RuntimeWarning: Mean of empty slice
  metrics["mean_iou"] = np.nanmean(iou)
/root/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--mean_iou/08bc20f4f895f3caf75fb9e3fada1404bded3c3265243d05327cbb3b9326ffe9/mean_iou.py:263: RuntimeWarning: Mean of empty slice
  metrics["mean_accuracy"] = np.nanmean(acc)
Trainer is attempting to log a value of "[nan]" of type <class 'list'> for key "eval/per_category_iou" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute.
Trainer is attempting to log a value of "[nan]" of type <class 'list'> for key "eval/per_category_accuracy" as a scalar. This invocation of Tensorboard's writer.add_scalar() is incorrect so we dropped this attribute.I use the same compute_metrics function as second link.So, I could not train the model for my scenario and I got some points that I could not figured out.I’m not sure what my num_labels parameter should be, 1 or 2.SegFormer image processor turn the masks (labels) into 0 or 255 values. Why is it not 0-1. Wouldn’t it be easier for classification?There is a parameter calledignore_indexin metrics.compute part. I could not understand very well, is it possible to be related the errors?In the first link, they are using nvidia/mit-b0 for segformer fine tuning but when I check the models on hugginface/models, it is used for image classification and there is seperatenvidia/segformer-b0-finetuned-ade-512-512. What is the reason that they did not used directly segformer-b0?If segformer is not suitable for binary segmentation, what else can I used for this case (better to be robust ones)?I did not find any configuration for loss function also. Is there anything about this I have to do?So, I’m struggling with this case for 2 days and I need some help. What should I do to use segformer in binary segmentation cases? Thanks in advance
2023-09-27T11:42:39Z
Hi,I had this issue a couple of days ago. If you care about the performance of the background class labels be sure to set thereduce_labelsargument to False and furthermore be sure to set ignore_index to 255 in themetric.compute()method, so that it looks like thismetrics = metric.compute(
            predictions=pred_labels,
            references=labels,
            num_labels=len(id2label),
            ignore_index=255,
            reduce_labels=image_processor.do_reduce_labels,
        )In the example scriptignore_indexis set to 0 and this gives you the NaN values in your metrics since you ignore the background class this way.