Optimize AND quantize with Optimum
Hi everyone,I would like to optimize AND quantize a NER model I have fine-tuned. I see that with the Optimum library it is easy to do both things separately, but I haven’t still managed to optimize and quantize the same model. Back with ONNXRuntime it was possible and easy to do. How can this be done with Optimum?Thanks!!
2022-09-27T12:10:09Z
You can check out this blog post:Optimizing Transformers with Hugging Face Optimumor the documentation:🤗 Optimum
2022-09-27T14:20:09Z
Hi Phil,Thank you for your answer. The first blog post seems to solve my question. Anyways, I think it uses an old version of Optimum that has some attributes that actual ones don’t. When exporting the optimized/quantized model, in the blog post they use theexportattribute. But, when I do the same (using version 1.4.0), it gives me an error saying'ORTOptimizer' object has no attribute 'export'. I have searched in the documentation but I haven’t find anything similar to the article for more recent versions.Thanks again!!
2022-09-27T18:17:05Z
Take a look at the documentation there are the API changes included for version 1.4.0, e.g. for quantization here:Quantization
2022-09-28T15:00:30Z
The problem I see with the latest version (1.4.0) compared to older versions is that it doesn’t have anexportattribute implemented nor anything similar that could optimize a quantized model or quantize an optimized model.As far as I understand, in version 1.4.0, it only allows you to go frommodel.onnxtomodel-optimized.onnxor frommodel.onnxtomodel-quantized.onnx, but then you can’t quantizemodel-optimized.onnxor optimizemodel-quantized.onnxto get amodel-optimized-quantized.onnx, while in other versions, withexportyou could (such as in the blog post you linked).Will be possible to do this in future versions? I think having a model quantized and optimized was such a good solution as it reduced a lot of space and achieved almost similar results as the original model.Thanks!!
2022-09-28T15:13:45Z
@jorgealro, that is already possible and supported you can provide thefile_namewhen either loading aORTModelor creating aOptimizer/Quantizer. This is explained and document in our documentation.
2022-09-29T15:50:33Z
Yes, that works for meThank you very much!!
2023-03-27T10:27:20Z
Hi@jorgealro,could you please explain how you solved the'ORTOptimizer' object has no attribute 'export'issue? I’m facing the same problem trying to optimize a CrossEncoder with optimum 1.7.3.Many thanks in advance!
2023-03-27T11:10:43Z
I found the solution, it might be useful to someone until thedocumentationgets updated (currently available is for v1.3.0).As of optimum==1.7.3, you should use theoptimizemethod, instead of theexportone:optimizer = ORTOptimizer.from_pretrained('model_name_or_path')
optimizer.optimize(
    save_dir='output_folder'
)
2023-03-27T21:30:36Z
Hi@f-dig, the documention seem to have moved:OptimizationSorry for the dead link!
2023-08-27T13:04:43Z
Had the same problem… and a bunch of other issues from the tutorial:Accelerated Inference with Optimum and Transformers Pipelines…Easy way to solve… delete all the argument, run, and use the logged comments (CLI messages) to find out the correct arguments.
2024-02-10T09:12:28Z
I managed to make the example from the blogAccelerated Inference with Optimum and Transformers Pipelineswork with the advise from this thread, but since this week the code breaks when quantizing the modelwith this errorRuntimeError: Unable to find data type for weight_name='/roberta/encoder/layer.0/attention/output/dense/MatMul_output_0'I have created a new post to explain the steps how to reproduce the errorOptimum library optimization and quantization fails - #2 by ddahlmeier