Loading only pre-trained backbone for Mask2Former
I am using huggingface Mask2Former model and would like toonlyload the pre-trained Swin backbone weights.Here is what I am currently doing.# Load pre-trained configuration (random initialization of weights)
model_config = Mask2FormerConfig.from_pretrained(
    "facebook/mask2former-swin-tiny-cityscapes-semantic"
)

# Cannot perform the below operation as backbone_config and backbone cannot be set
# model_config.use_pretrained_backbone = True
# model_config.backbone = "microsoft/swin-tiny-patch4-window7-224"

# So override the backbone config
model_config.backbone_config = None
model_config.use_pretrained_backbone = True
model_config.backbone = "microsoft/swin-tiny-patch4-window7-224"The above code results in an error w.r.t shape mismatch as self.encoder.channels in Mask2FormerPixelDecoder object expects…(hidden_states_norms): ModuleDict((stage1): LayerNorm((96,), eps=1e-05, elementwise_affine=True)(stage2): LayerNorm((192,), eps=1e-05, elementwise_affine=True)(stage3): LayerNorm((384,), eps=1e-05, elementwise_affine=True)(stage4): LayerNorm((768,), eps=1e-05, elementwise_affine=True))…How do i ensure that only the backbone has pre-trained weights as opposed to the other blocks like pixel decoder and transformer decoder?PS:model_config = Mask2FormerConfig.from_pretrained(
    "facebook/mask2former-swin-tiny-cityscapes-semantic"
)
# Loading pre-trained configurations with randomly initilaized weights
model_config.backbone_config = SwinConfig.from_pretrained("microsoft/swin-tiny-patch4-window7-224", out_features=['stage1', 'stage2', 'stage3', 'stage4'])
# The above method works when matching the dimensions but i cannot load the pre-trained weights.