Convert DeBERTa model to ONNX with mixed precision
Hello. I’m usingdeberta-v3-basefor a text classification task. After training I’m converting a pytorch model to ONNX format like this:model_kind, model_onnx_config = FeaturesManager.check_supported_model_or_raise(
        model, feature="sequence-classification"
    )
    onnx_config = model_onnx_config(config)
    onnx_inputs, onnx_outputs = export(
        preprocessor=tokenizer,
        model=model,
        config=onnx_config,
        opset=15,
        output=Path(args.output_model) / "onnx_model.onnx",
    )Everything works like a charm except that the size of the model is twice the size of the original DeBERTa - ~750MB. Because of it I want to convert it with mixed precision, i.e. fp16. I tried two approaches:Runmodel.half()before ONNX conversionUse the following code:from onnxruntime.transformers import optimizer
optimized_model = optimizer.optimize_model("onnx_model.onnx", model_type='bert', num_heads=12, hidden_size=768, use_gpu=False, opt_level=0)
optimized_model.convert_float_to_float16()
optimized_model.save_model_to_file("onnx_model_fp16.onnx")But in both cases I get this error during inference on CPU:2023-01-06 10:46:46.332352649 [W:onnxruntime:, constant_folding.cc:179 ApplyImpl] Could not find a CPU kernel and hence can't constant fold LayerNormalization node 'LayerNorm_1'
2023-01-06 10:46:46.414666254 [W:onnxruntime:, constant_folding.cc:179 ApplyImpl] Could not find a CPU kernel and hence can't constant fold LayerNormalization node 'LayerNorm_1'
2023-01-06 10:46:46.425605272 [W:onnxruntime:, constant_folding.cc:179 ApplyImpl] Could not find a CPU kernel and hence can't constant fold LayerNormalization node 'LayerNorm_1'I also tried to setuse_gpu=Truein optimize_model method. Errors disappeared, but the inference time was 3-4 time slower.