:rocket: Optimum Transformers: accelerated NLP pipelines with Infinity speed
Optimum TransformersAccelerated NLP pipelines for fast inferenceon CPU and GPU. Built withTransformers, Optimum and ONNX runtime.DisclaimerThis project is my inspiration ofHuggingface Infinity. And first step done bySuraj Patil.@huggingface’s pipeline API is awesome!, right? And onnxruntime is super fast !. Wouldn’t it be great to combine these two?– Tweet by Suraj PatilIt was under this slogan that I started doing this project!And the main goal was to show myself to get into@huggingfaceteamHow to useQuick startThe usage is exactly the same as original pipelines, except minor improves:from optimum_transformers import pipeline

pipe = pipeline("text-classification", use_onnx=True, optimize=True)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]use_onnx- converts default model to ONNX graphoptimize- optimizes converted ONNX graph withOptimumOptimum configReadOptimumdocumentation for more detailsfrom optimum_transformers import pipeline
from optimum.onnxruntime import ORTConfig

ort_config = ORTConfig(quantization_approach="dynamic")
pipe = pipeline("text-classification", use_onnx=True, optimize=True, ort_config=ort_config)
pipe("This restaurant is awesome")
# [{'label': 'POSITIVE', 'score': 0.9998743534088135}]BenchmarkWith notebookYou can benchmark pipelines easier withbenchmark_pipelinesnotebook.With own scriptfrom optimum_transformers import Benchmark

task = "sentiment-analysis"
model_name = "philschmid/MiniLM-L6-H384-uncased-sst2"
num_tests = 100

benchmark = Benchmark(task, model_name)
results = benchmark(num_tests, plot=True)ResultsNote: These results were collected on my local machine. So if you have high performance machine to benchmark, please contact mesentiment-analysisAlmost the same as inInifinity launchvideoAWS VM: g4dn.xlargeGPU: NVIDIA T4128 tokens2.6 msResulting plot1280×960 31.4 KBzero-shot-classificationWithtypeform/distilbert-base-uncased-mnliResulting plot1280×960 31.4 KBtoken-classificationResulting plot1280×960 35.9 KBResulting plot1280×960 31.6 KBMore results are available in project repository:GitHub.
2022-03-25T17:10:27Z
This is awesome!! You did exactly what I wanted to achieve some weeks ago when I wrote a blog post about a custom way to use pipelines with onnx. But it wasn’t efficient because it was also loading PyTorch model along with onnx. But your solution looks awesome!!!I want to contribute!!Congratz
2022-03-25T17:43:44Z
Thank you@ChainYo! I really appreciate thisHope it will be useful for all the community!
2022-03-25T17:52:26Z
Hugging Face Team, may be it is time to add new discussion topics here: ONNX, Optimum?
2022-03-25T20:21:22Z
Indeed, created a category forOptimum, thanks for the suggestion!