Exporting UDOP to ONNX fails
When trying to export the UDOP model to ONNX using torch.onnx.export(), the export fails. Below is the output from running transformers-cli env:transformersversion: 4.39.3Platform: Linux-5.4.0-1072-aws-x86_64-with-glibc2.29Python version: 3.10.13Huggingface_hub version: 0.21.4Safetensors version: 0.4.2Accelerate version: 0.28.0Accelerate config:    not foundPyTorch version (GPU?): 2.2.2+cu118 (True)Tensorflow version (GPU?): not installed (NA)Flax version (CPU?/GPU?/TPU?): not installed (NA)Jax version: not installedJaxLib version: not installedUsing GPU in script?: NoUsing distributed or parallel set-up in script?:NoI have included here a minimal code example to reproduce the issue:from PIL import Image
from torch.onnx import export
from transformers import (
    UdopEncoderModel,
    UdopProcessor,
)

# Load UDOP processor
udop_processor = UdopProcessor.from_pretrained("microsoft/udop-large", apply_ocr=False)
# Create dummy input for ONNX export
dummy_encodings = udop_processor(
    images=Image.new(mode='RGB', size=(224, 224)),
    text=['dummy', 'text'],
    boxes=[[0, 0, 100, 100], [750, 750, 850, 850]],
    return_tensors="pt",
    max_length=512,
    padding="max_length",
)
# Load base UDOP model from HuggingFace hub
udop_model = UdopEncoderModel.from_pretrained("microsoft/udop-large")
# Export model to ONNX using torch.onnx.export()
export(
    model=udop_model,
    args=(
        dummy_encodings.input_ids,
        dummy_encodings.bbox,
        dummy_encodings.attention_mask,
        dummy_encodings.pixel_values,
    ),
    f='model.onnx',
)Running this example produces the following error:/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:399: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if height != self.image_size[0] or width != self.image_size[1]:
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:339: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  torch.arange(len(ocr_points))[:, None].repeat(1, ocr_points.size(-1))[:, :, None].to(ocr_points),
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:345: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  rows, cols = zip(*ind)
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:345: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  rows, cols = zip(*ind)
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:348: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  input_vision_patches = [image_embeddings[i][patch_inds[i]] for i in range(len(patch_inds))]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:355: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  visual_bbox = [visual_bbox[i][patch_inds[i]] for i in range(len(patch_inds))]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:357: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  visual_attention_mask = [torch.tensor([1] * len(item)).to(attention_mask) for item in visual_bbox]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:357: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  visual_attention_mask = [torch.tensor([1] * len(item)).to(attention_mask) for item in visual_bbox]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:293: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if m > 0:
Traceback (most recent call last):
  File "src/pd_ai_data_extractor/scripts/udop_to_onnx.py", line 24, in <module>
    export(
  File "/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1613, in _export
    graph, params_dict, torch_out = _model_to_graph(
  File "/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1135, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args)
  File "/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 1011, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/venv/lib/python3.10/site-packages/torch/onnx/utils.py", line 915, in _trace_and_get_graph_from_model
    trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
  File "/venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 1296, in _get_trace_graph
    outs = ONNXTracedModule(
  File "/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/venv/lib/python3.10/site-packages/torch/jit/_trace.py", line 138, in forward
    graph, out = torch._C._create_graph_by_tracing(
RuntimeError: 0 INTERNAL ASSERT FAILED at "../torch/csrc/jit/ir/alias_analysis.cpp":615, please report a bug to PyTorch. We don't have an op for aten::full_like but it isn't a special case.  Argument types: Tensor, bool, NoneType, NoneType, NoneType, bool, NoneType,

Candidates:
        aten::full_like(Tensor self, Scalar fill_value, *, ScalarType? dtype=None, Layout? layout=None, Device? device=None, bool? pin_memory=None, MemoryFormat? memory_format=None) -> Tensor
        aten::full_like.out(Tensor self, Scalar fill_value, *, MemoryFormat? memory_format=None, Tensor(a!) out) -> Tensor(a!)I have tracked this issue down to line 336 of transformers/models/udop/modeling_udop.py:patch_inds = torch.full_like(image_embeddings[:, :, 0], True).bool()This issue can be corrected by replacing this line with:patch_inds = torch.full_like(image_embeddings[:, :, 0], 1).bool()However, after making that change, there is a new issue with the following traceback:/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:399: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if height != self.image_size[0] or width != self.image_size[1]:
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:339: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  torch.arange(len(ocr_points))[:, None].repeat(1, ocr_points.size(-1))[:, :, None].to(ocr_points),
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:345: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  rows, cols = zip(*ind)
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:345: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  rows, cols = zip(*ind)
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:348: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  input_vision_patches = [image_embeddings[i][patch_inds[i]] for i in range(len(patch_inds))]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:355: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  visual_bbox = [visual_bbox[i][patch_inds[i]] for i in range(len(patch_inds))]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:357: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  visual_attention_mask = [torch.tensor([1] * len(item)).to(attention_mask) for item in visual_bbox]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:357: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  visual_attention_mask = [torch.tensor([1] * len(item)).to(attention_mask) for item in visual_bbox]
/venv/lib/python3.10/site-packages/transformers/models/udop/modeling_udop.py:293: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if m > 0:
/venv/lib/python3.10/site-packages/torch/onnx/utils.py:1703: UserWarning: The exported ONNX model failed ONNX shape inference. The model will not be executable by the ONNX Runtime. If this is unintended and you believe there is a bug, please report an issue at https://github.com/pytorch/pytorch/issues. Error reported by strict ONNX shape inference: [ShapeInferenceError] Inference error(s): (op_type:Add, node name: /encoder/Add_3): [ShapeInferenceError] Incompatible dimensions
(op_type:Concat, node name: /encoder/Concat_9): [TypeInferenceError] Input 0 expected to have type but instead is null
(op_type:Add, node name: /encoder/Add_4): [TypeInferenceError] Input 0 expected to have type but instead is null
 (Triggered internally at ../torch/csrc/jit/serialization/export.cpp:1484.)
  _C._check_onnx_proto(proto)I have not been able to track down the cause of this new issue so any help with this would be much appreciated.I addition submitted a bug report to PyTorch atONNX export fails for aten::full_like op when exporting UDOP model from transformers · Issue #122898 · pytorch/pytorch · GitHub. There are some suggestions in that issue but none that resolved the underlying bug since it seems to be an issue with the UDOP implementation in transformers.