Optimum arm64 quantized models on Apple Silicon (M1)
Hi everyone,I’m running some quantization experiments on my MBP 13 M1 with optimum and onnx. I see that arm64 quantized model use 2x less time for an inference pass than basic ORT model without quantization. At the same time, if I ran avx2-quantized model, it also uses less time for an inference pass, but it is not as fast as arm64 model(see screenshot). Does optimum actually uses arm64 instructions during inference of arm64-quantized models? Or the speedup is just a result of some other default optimizations?Screenshot 2023-02-16 at 22.36.131132×486 46.2 KB
2023-05-03T11:45:58Z
Hi@Rexhaif, sorry for the late responseThe only difference between thearm64 quantization configand theavx2 oneis that weights are mapped to signed 8-bit integers for arm64 and unsigned 8-bit integers for avx2. So I guess this fits better the M1.