Onnx export functionality failure for facebook/opt-2.7b with optimum CLI
steps followed:Load facebook torch modelApply smooth quantizationSave pytorch modelimport torchimport argparsefrom transformers import AutoTokenizer, OPTForCausalLMimport osimport smoothfrom optimum.onnxruntime import ORTQuantizerfrom optimum.onnxruntime.configuration import AutoQuantizationConfigimport onnximport onnxruntime as ortimport numpy as npimport timeimport stringimport randomif__name__ == “main”:parser = argparse.ArgumentParser()parser.add_argument(“–model_name”, help=“Different OPT model sizes”, type=str, default=“opt-2.7b”)parser.add_argument(“–onnx”, help=“load model from huggingface, smoothquant and save onnx”, action=‘store_true’)args = parser.parse_args()if args.onnx:
    model = OPTForCausalLM.from_pretrained("facebook/" + args.model_name)
    tokenizer = AutoTokenizer.from_pretrained("facebook/" + args.model_name)
    model.tokenizer = tokenizer 
    
    act_scales = torch.load(os.getenv("PYTORCH_PATH") + "/smoothquant/act_scales/" + "%s.pt"%args.model_name)
    smooth.smooth_lm(model, act_scales, 0.5)
    print(model)
    
    prompt = ''.join(random.choices(string.ascii_lowercase + " ", k=model.config.max_position_embeddings))
    #inputs = tokenizer(prompt, return_tensors="pt")  # takes a lot of time
    inputs = tokenizer("What is meaning of life", return_tensors="pt") 
    print(f"inputs: {inputs}")
    print(f"inputs.input_ids: {inputs.input_ids}")
    for key in inputs.keys():
        print(inputs[key].shape)
        print(inputs[key])
    model_out = model(inputs.input_ids)
    print(f"{(model_out.logits.shape)=}")
    out_dir = "./%s_smoothquant"%args.model_name
    if not os.path.exists(out_dir):
        os.makedirs(out_dir)
    model.save_pretrained(out_dir+"/pytorch")Use optimum cli to convert from pyorch to onnx modeloptimum-cli export onnx -m opt-2.7b_smoothquant\pytorch --task text-generation-with-past opt-2.7b_smoothquant\onnx  --framework pt --no-post-processI am seeing this warning in the log“The ONNX export succeeded with the warning: The maximum absolute difference between the output of the reference model and the ONNX exported model is not within the set tolerance 1e-05”please find the below log:logits: max diff = 14.66848087310791present.8.key: max diff = 1.1682510375976562e-05present.18.key: max diff = 7.810757637023926present.18.value: max diff = 3.938399076461792present.19.key: max diff = 5.7716522216796875present.19.value: max diff = 3.3612823486328125present.20.key: max diff = 3.3796701431274414present.20.value: max diff = 2.428762435913086present.21.key: max diff = 3.4451732635498047present.21.value: max diff = 2.12455153465271present.22.key: max diff = 3.651796817779541present.22.value: max diff = 2.4826853275299072present.23.key: max diff = 4.581423759460449present.23.value: max diff = 2.5400819778442383present.24.key: max diff = 10.186495780944824present.24.value: max diff = 5.759485244750977present.25.key: max diff = 10.965536117553711present.25.value: max diff = 6.26738977432251present.26.key: max diff = 2.7324962615966797present.26.value: max diff = 2.6341564655303955present.27.key: max diff = 4.848768711090088present.27.value: max diff = 5.190319061279297present.28.key: max diff = 5.36247444152832present.28.value: max diff = 5.969284534454346present.29.key: max diff = 5.126808166503906present.29.value: max diff = 6.3877997398376465present.30.key: max diff = 4.772220134735107present.30.value: max diff = 5.988787651062012present.31.key: max diff = 5.070601940155029present.31.value: max diff = 6.673184394836426The ONNX export succeeded with the warning: The maximum absolute difference between the output of the reference model and the ONNX exported model is not within the set tolerance 1e-05:logits: max diff = 7.611649513244629present.18.key: max diff = 3.4201674461364746present.18.value: max diff = 2.020570755004883present.19.key: max diff = 2.3315491676330566present.19.value: max diff = 1.6240568161010742present.20.key: max diff = 1.7808356285095215present.20.value: max diff = 1.4954586029052734present.21.key: max diff = 1.8739492893218994present.21.value: max diff = 1.7475067377090454present.22.key: max diff = 2.3744964599609375present.22.value: max diff = 1.9387221336364746present.23.key: max diff = 2.560657024383545present.23.value: max diff = 2.3288230895996094present.24.key: max diff = 8.912554740905762present.24.value: max diff = 5.827574253082275present.25.key: max diff = 10.221282958984375present.25.value: max diff = 4.572129726409912present.26.key: max diff = 2.0476551055908203present.26.value: max diff = 2.327301502227783present.27.key: max diff = 3.158855676651001present.27.value: max diff = 3.6903629302978516present.28.key: max diff = 4.461582660675049present.28.value: max diff = 4.333994388580322present.29.key: max diff = 4.099486351013184present.29.value: max diff = 5.0107622146606445present.30.key: max diff = 3.9387893676757812present.30.value: max diff = 5.12119197845459present.31.key: max diff = 3.750337600708008present.31.value: max diff = 4.961455821990967.The exported model was saved at: opt-2.7b_smoothquant/onnx