Potential Memory Leak for ORTModelForCausalLM with TensorRT Providor
DescriptionHi there,first, thank you for the optimum library, it really works well. However, i may found a potential memory leak while working with ORTModelForCausalLM with TensorRT Provider.We follow closely follow Optimum TensorRT Guide from User Guide Docs and run the following code:session_options = onnxruntime.SessionOptions()
session_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_DISABLE_ALL
provider_options = {
        "trt_engine_cache_enable": True,
        "trt_engine_cache_path": cache_path,
        "trt_int8_enable": True }
tok = AutoTokenizer.from_pretrained(onnx_path, use_fast=True)
ort_model = ORTModelForCausalLM.from_pretrained(
      onnx_path,
      export=False,
      provider="TensorrtExecutionProvider",
      use_cache=False,
      session_options=session_options,
      provider_options=provider_options,
      log_severity_level=0)Using this snippet, the onnx model is converted to an engine (probably with trtexec in background), however, it requires a massive memory load. (More than 125GB ram for a 350 Million parameter model). Is there any reason for that or is that expected?EnvironmentDocker Image:nvcr.io/nvidia/pytorch:22.12-py3TensorRT Version            8.501Python                              3.8.10Pips:transformers                   4.26.1optimum                          1.7.1onnx                                 1.12.0onnxruntime-gpu          1.14.1pytorch-quantization    2.1.2pytorch-triton                 2.0.0+b8b470bc59torch                                 1.13.1torch-tensorrt                 1.3.0torchtext                          0.13.0a0+fae8e8ctorchvision                       0.15.0a0Steps To ReproduceConvert the model to transformer model to onnx via optimum-cli.Do the quantization (we did it exactly) like inquantization guideRun Infer_shape.py as suggested by trtexec.Run the code from aboveThank you very much for any help!
2023-05-03T11:59:13Z
Hi@cdawg, that does sound like a lot of memory for a 350M-parameter model!Have you got the chance to compare with a native TRT way of building the engine (i.e. without relying on ORT’s TRT backend)?
2023-05-09T07:15:18Z
Hi Regisss,yes i tried it with much lower memory requirements. It gets a little bit better if i use workspace size, but it does that the memory requirements are bounded by the workspace size. Am i doing something wrong?
2023-05-09T07:33:30Z
Are you able to reproduce this issue with a model which is publicly accessible on the Hugging Face Hub? So that I can reproduce this issue and start investigating it in the next few days.
2023-06-02T08:49:11Z
Sorry for the late reponse. However, for me the problem is solved. because i switch to accelerate and bitsandbytes.The model of intrest iscodegen-350M-multi