Optimum & RoBERTa: how far can we trust a quantized model against its pytorch version?
Hi,I did adapt thiscodefromOptimum githubabout thesequence-classificationmodeldistilbert-base-uncased-finetuned-sst-2-englishto themasked-lmmodelRoBERTa base.It works (see the code below) but the predictions are different (see results at the end of this post). I understand the reason (quantization changes the weights values and then the forward calculation of a prediction) but how far does it change the model for inference?My question is:how far can we trust a quantized model against its pytorch version?My code:Quantization of the model!pip install datasets
!python -m pip install git+https://github.com/huggingface/optimum.git#egg=optimum[onnxruntime]

from optimum.onnxruntime.configuration import AutoQuantizationConfig
from optimum.onnxruntime import ORTQuantizer

# The model we wish to quantize
#model_checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
model_checkpoint = "roberta-base"

# The type of quantization to apply
qconfig = AutoQuantizationConfig.arm64(is_static=False, per_channel=False)

#quantizer = ORTQuantizer.from_pretrained(model_checkpoint, feature="sequence-classification")
quantizer = ORTQuantizer.from_pretrained(model_checkpoint, feature="masked-lm")

# Quantize the model!
quantizer.export(
    onnx_model_path="model.onnx",
    onnx_quantized_model_output_path="model-quantized.onnx",
    quantization_config=qconfig,
)Inference (get logits)from functools import partial
from datasets import Dataset
from optimum.onnxruntime.model import ORTModel

# Load quantized model
ort_model = ORTModel("model-quantized.onnx", quantizer._onnx_config)

# Create a dataset or load one from the Hub
#ds = Dataset.from_dict({"sentence": ["I love burritos!"]})
sentence = "The goal of life is <mask>."
ds = Dataset.from_dict({"sentence": [sentence]})

# Tokenize the inputs
def preprocess_fn(ex, tokenizer):
    return tokenizer(ex["sentence"])

tokenized_ds = ds.map(partial(preprocess_fn, tokenizer=quantizer.tokenizer))
ort_outputs = ort_model.evaluation_loop(tokenized_ds)

# Extract logits!
logits = ort_outputs.predictionsPredictionimport numpy as np

tokens = np.array(tokenized_ds['input_ids'])
mask_token_index = (tokens == quantizer.tokenizer.mask_token_id)[0].nonzero()[0][0]
predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)

# replace mask token by predicted token
tokens[0][mask_token_index] = predicted_token_id

print(sentence)
print(quantizer.tokenizer.decode(tokens[0], skip_special_tokens=True))ResultsThe previous code will print:The goal of life is <mask>.
The goal of life is life.However, the pytorch version of RoBERTa base either in thehub, either inSpaces, gives as result:image1483×844 29.8 KB
2022-05-31T18:20:13Z
Hi.I did compare inference time of Pytorch and Optimum RoBERTa base model in thisColab notebook.Here the results:Pytorch: 247 msOptimum: 719.0 msSomething wrong, no?
2022-06-02T12:54:32Z
Hi@pierreguillou,ORTModelwas a class added with the primary function for users to be able to obtain the evaluation results in order to compare their original model with the resulting quantized / optimized model, and was meant to be temporary. This class is now depreciated and we encourage users to use our new classesORTModelForXxx, which will allow you to get better perfomances.pierreguillou:My question is:how far can we trust a quantized model against its pytorch version?The best way to evaluate your resulting quantized model is to use the same evaluation as its full-precision counterpart and compare both. I would say that there is no possibility to know until then, as different type of quantization (dynamic vs static for example) would impact the model differently, also some architectures and given tasks could be more sensitive to quantization.
2022-06-02T14:50:41Z
Hi@echarlaix,thanks for the link to new classesORTModelForXxxbut the second code example does not work in Colab (see screen shot). This is the code about:Optimum Inference also includes methods to convert vanilla Transformers models to optimized ones. Simply passfrom_transformers=Trueto thefrom_pretrained()method, and your model will be loaded and converted to ONNX on-the-fly.image2144×1432 254 KBAnd there is no link to a notebook in the following sentence (the sentence comes just before the title “Working with the Hugging Face Model Hub”):You can find a complete walkhrough Optimum Inference for ONNX Runtime in this [notebook](todo:add-link).Great if you can help on these issues.
2022-06-02T15:31:20Z
The following code is working on my side (adapted from thedocumentation), can you try :model = ORTModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english", from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english")
onnx_classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)
text = "Hello, my dog is cute"
pred = onnx_classifier(text)
2022-06-02T21:55:23Z
Yes, your code is working@echarlaixbecause you did not writetext=in the pipeline objectonnx_classifieras it is done in the Optimum page (checkSwitching from Transformers to Optimum Inferenceand the screen shot below).optimum3054×1668 460 KB
2022-06-03T13:01:08Z
Good catch, I will correct the documentation, thanks !
2022-06-03T16:25:12Z
Thank you@echarlaix.Another question: I do not see in thedoc, the classORTModelForMaskedLM(onlyORTModelForCausalLM).Do you plan to update the Optimum library with (at least) this class?
2022-06-06T08:39:31Z
Hi@pierreguillouYes we plan to add many moreORTModelForXxxin the future, and are currently working onORTModelForSeq2SeqLM(#199). Everyone is welcomed to contribute so don’t hesitate to open a PR if you find time to work on the integration ofORTModelForMaskedLM.
2022-07-22T02:37:58Z
Hi There, I tried the above code snippet and also installed the necessary packages but got a symlink error below:image1413×1038 138 KBI tried turning on the dev mode on my windows PC and did a restart but still same error. It only works in Colab for me.
2022-07-27T08:16:19Z
Let’s move this discussion toSymlink error when importing ORTSeqClass model via Pipelineas you are describing the same problem there.