ORT CLI vs. Programmatic
Hi,As an experiment, I am running a conversion/optimization from PyTorch to ONNX with dolly-v2-3b. The CLI approach ran successfully for optimization O3 on a Dell 7540 (Xeon E-2286M x 16; 64GB memory). It resulted in a 15% reduction when compared to a not optimized conversion.When attempting the same via a programmatic approach the program gets killed by the Kernel as it exceeds available memory resources (100% RAM and Swap). This happens when the optimze() method of ORTOptimizer is called. The last output seen is “Optimizing model…”. The configuration was generated with AutoOptimizationConfigO3().With the CLI execution, memory use never exceeds 80%, with about 25% of Swap. Hence, I have a few questions:Is there a guide as to required resources for programmatic and CLI ORT Optimum execution?What makes it require more resources under the API based optimization?In both cases optimization is the same with Level O3, is there more optimization techniques applied with the programmatic approach as compared to the CLI approach?Thanks,Borell
2023-07-13T18:54:48Z
Hi Borell! Could you share the CLI commands and the script you used to compare both approaches please?
2023-07-13T20:48:25Z
Hi regisss,Thanks for the reply, I can certainly send you the code. I wrote a Python module which runs CLI or optionally programmatic code. Doesn’t seem that there is an option to attach files. I can paste the code in a reply if that is OK, it is not very long.Sorry my first time using this forum.BBorell
2023-07-14T07:30:14Z
No worries, it’s even better to directly share the code here surrounding your code snippets with ```
2023-07-14T13:17:39Z
regisss,Here is the code. To run with CLI, change cliBool = False to True and change the location path “model_save”. Comparison of size is then done manually by verifying the size of files created at path model_save.The ultimate objective is not Dolly but something like a Flan-T5 XXX. I am exploring the feasibility of using ORT. I attempted earlier with OpenVINO, but their implementation crashed. It is now being evaluated by Intel developers after they confirmed the issue with their own sample code.‘’’#!/usr/bin/env python#standardimportsimport osimport subprocessfrom pathlib import Path#third-partyimportsfrom transformers import AutoTokenizerfrom optimum.onnxruntime import  ORTModelForCausalLMfrom optimum.pipelines import pipelineimport torchfrom tensorflow.python.ops.initializers_ns import variables#fromoptimum.onnxruntime import  OptimizationConfig, ORTOptimizer#deferredto optimizationcliBool = False#gettingfromhuggingface.comodel_remote = ‘databricks/dolly-v2-3b’model_path = Path(‘~/.cache/huggingface/hub/models–databricks–dolly-v2-3b’).expanduser()if model_path.exists():print(“************* 1: Getting local PyTorch Model”)#… from local cache; process does not expand user path; also, needs to point at config.json file locationmodel_path = model_path / ‘snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df’#savingto: /home/jellybean/.cache/huggingface/hub/models–databricks–dolly-v2-3b/snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df/optimum/onnx_rtmodel_save = model_path / ‘optimum/onnx_rt_optimized’else:print(“************* 1: Getting Hugging Face PyTorch Model”)model_path = model_remote#mannuallymove it to cache snapshot upon createdmodel_save = Path(‘~/.cache/huggingface/hub/’).expanduser() / ‘optimum/onnx_rt’GPU not enough memory: the following reports before onnx conversion starts#mem= torch.cuda.mem_get_info()#print(f"CUDA Memory - Available: {mem[0]} Total: {mem[1]}“)#print(f"CUDA Memory - Total: {torch.cuda.get_device_properties(0).total_memory} Reserved: {torch.cuda.memory_reserved(0)} Allocated: {torch.cuda.memory_allocated(0)}”)#print(f"{torch.cuda.memory_summary(device=0, abbreviated=True)}")#checkfile exist so we don’t convert againconfig = model_save / ‘config.json’‘’‘torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 76.00 MiB…… (GPU 0; 5.79 GiB total capacity; 5.26 GiB already allocated; 69.50 MiB free;…… 5.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory…… try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF…… but os.environ[‘PYTORCH_CUDA_ALLOC_CONF’] = ‘max_split_size_mb:21’ still gets OutOfMemoryErrorcli = ‘optimum-cli export onnx -m {model_path} --cache_dir {model_save} --task text-generationcli += f’–framework pt --device cuda --optimize O4 --batch_size 32  {model_save}’Cause: --device cuda --optimize O4‘’’if not config.exists():if cliBool:print(“************* 2: CLI-ONNX Convert/Optimize”)#switchingto CPU; need to specify --task if not from huggingface; optimizing to Level O3…#… export dolly to onnx: > optimum-cli export onnx --helpcli = f’optimum-cli export onnx -m {model_path} --task text-generation-with-past --framework pt --no-post-process --optimize O3 {model_save}’with subprocess.Popen([cli], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE) as proc:_, _ = proc.communicate()else:print(“************* 2: Programmatic-ONNX Converte/Optimize ONNX”)from optimum.onnxruntime import AutoOptimizationConfig, ORTOptimizerprint("************* 2.a")
	model = ORTModelForCausalLM.from_pretrained(model_path, export=True)
	print("************* 2.b")
	optimizer = ORTOptimizer.from_pretrained(model)
	print("************* 2.c")
	#optimizing to Level O3
	optimization_config = AutoOptimizationConfig.O3()
	print(optimization_config)
	raise SystemExit
	print("************* 2.d")
	optimizer.optimize(save_dir=model_save, optimization_config=optimization_config)else: print(“************* 2: Skipping ONNX Convertion/Optimization”)#inferenceusing pt to onnx modelprint(“************* 3: ONNX Model Inference”)tokenizer = AutoTokenizer.from_pretrained(model_save)model_ort =  ORTModelForCausalLM.from_pretrained(model_save)ort_pipe = pipeline(‘text-generation’, model=model_ort, tokenizer=tokenizer, accelerator=‘ort’, framework=‘pt’, device=-1, model_kwargs={“load_in_8bit”: True})#try4 different promptsprompt1 = [‘Explain to me what is love.’, ‘You are an idiot’]#producesa list of two lists with dictsprompt2 = [‘Explain to me what is hate.’]#producesa list of one list with dictprompt3 = ‘Do you love yourself?.’#producesa list with a dictprompt4 = ‘I hate you!’#producesa list with a dictprompts = [prompt1, prompt2, prompt3, prompt4]for prompt in prompts:res = ort_pipe(prompt)print(f"Prompt: {prompt}“)for item in res:next_item = itemwhile isinstance(next_item, list): next_item = next_item[0]#loopuntil no list (looking for dict)print(f”\n{next_item[‘generated_text’]}")else: passelse: pass‘’’
2023-07-14T13:38:07Z
Here is the code again without getting mangled up.‘’’#!/usr/bin/env python#standardimportsimport osimport subprocessfrom pathlib import Pathdevs = subprocess.Popen(‘lspci -nnk | grep -iA2 vga’, shell=True, stdout=subprocess.PIPE).stdout.read().decode(‘utf-8’).split(‘\n’)print()for dev in devs: print(‘Device:’, dev)print(“Info on devices not being used.\n”)#third-partyimportsfrom transformers import AutoTokenizerfrom optimum.onnxruntime import  ORTModelForCausalLMfrom optimum.pipelines import pipelineimport torchfrom tensorflow.python.ops.initializers_ns import variables#fromoptimum.onnxruntime import  OptimizationConfig, ORTOptimizer#deferredto optimizationcliBool = False#gettingfromhuggingface.comodel_remote = ‘databricks/dolly-v2-3b’model_path = Path(‘~/.cache/huggingface/hub/models–databricks–dolly-v2-3b’).expanduser()if model_path.exists():print(“************* 1: Gettting local PyTorch Model”)#… from local cache; process does not expand user path; also, needs to point at config.json file locationmodel_path = model_path / ‘snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df’#savingto: /home/jellybean/.cache/huggingface/hub/models–databricks–dolly-v2-3b/snapshots/f6c9be08f16fe4d3a719bee0a4a7c7415b5c65df/optimum/onnx_rtmodel_save = model_path / ‘optimum/onnx_rt_optimized’else:print(“************* 1: Getting Hugging Face PyTorch Model”)model_path = model_remote#mannuallymove it to cache snapshot upon createdmodel_save = Path(‘~/.cache/huggingface/hub/’).expanduser() / ‘optimum/onnx_rt’#checkfile exist so we don’t convert againconfig = model_save / ‘config.json’if not config.exists():if cliBool:print(“************* 2: CLI-ONNX Convert/Optimize”)#switchingto CPU; need to specify --task if not from huggingface; optimizing to Level O3…#… export dolly to onnx: > optimum-cli export onnx --helpcli = f’optimum-cli export onnx -m {model_path} --task text-generation-with-past --framework pt --no-post-process --optimize O3 {model_save}’with subprocess.Popen([cli], shell=True, stdout=subprocess.PIPE, stdin=subprocess.PIPE) as proc:_, _ = proc.communicate()else:print(“************* 2: Programmatic-ONNX Convert/Optimize ONNX”)from optimum.onnxruntime import AutoOptimizationConfig, ORTOptimizerprint("************* 2.a")
	model = ORTModelForCausalLM.from_pretrained(model_path, export=True)
	print("************* 2.b")
	optimizer = ORTOptimizer.from_pretrained(model)
	print("************* 2.c")
	#optimizing to Level O3
	optimization_config = AutoOptimizationConfig.O3()
	print(optimization_config)
	print("************* 2.d")
	optimizer.optimize(save_dir=model_save, optimization_config=optimization_config)else: print(“************* 2: Skipping ONNX Convertion/Optimization”)#inferenceusing pt to onnx modelprint(“************* 3: ONNX Model Inference”)tokenizer = AutoTokenizer.from_pretrained(model_save)model_ort =  ORTModelForCausalLM.from_pretrained(model_save)ort_pipe = pipeline(‘text-generation’, model=model_ort, tokenizer=tokenizer, accelerator=‘ort’, framework=‘pt’, device=-1, model_kwargs={“load_in_8bit”: True})#try3 different promptsprompt1 = [‘Explain to me what is love.’, ‘You are an idiot’]#producesa list of two lists with dictsprompt2 = [‘Explain to me what is hate.’]#producesa list of one list with dictprompt3 = ‘Do you love yourself?.’#producesa list with a dictprompt4 = ‘I hate you!’#producesa list with a dictprompts = [prompt1, prompt2, prompt3, prompt4]for prompt in prompts:res = ort_pipe(prompt)print(f"Prompt: {prompt}“)for item in res:next_item = itemwhile isinstance(next_item, list): next_item = next_item[0]#loopuntil no list (looking for dict)print(f”\n{next_item[‘generated_text’]}")else: passelse: pass‘’’
2023-07-21T11:46:50Z
Hi, a difference between the export CLI andORTModel.from_pretrained(..., export=True)is that by default the CLI fuses the decoder without and with KV cache.At the time, this feature was quite experimental and we did not make it default in thefrom_pretrainedexport.You can try to passuse_merged=Truetofrom_pretrainedto lower the memory usage.For reference:huggingface.coExport functionsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coModelsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.GitHubRelease v1.8: extended BetterTransformer support, ONNX merged seq2seq models...Extended BetterTransformer support
Various improvements in the PyTorch BetterTransformer integration.

[BT] add BetterTransformer support for ProphetNet by @hirotasoshu in #923
Improve bettertransf...See as well the note about the pytorch version here:Release v1.9: extended ONNX, ONNX Runtime support · huggingface/optimum · GitHub
2023-07-21T17:58:24Z
Hi fxmarty,Thanks for the advice. I added `use_merged=True. During the execution of from_pretrained(…) the following error shows:Traceback (most recent call last):File “/home/jellybean/workspace/careless_navigator/src/optimum/./dolly_ort_optimum.py”, line 87, inmodel = ORTModelForCausalLM.from_pretrained(model_path, export=True, use_merged=True)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File “/home/jellybean/.virtualenvs/careless_navigator/lib/python3.11/site-packages/optimum/onnxruntime/modeling_ort.py”, line 646, in from_pretrainedreturn super().from_pretrained(^^^^^^^^^^^^^^^^^^^^^^^^…File “/home/jellybean/.virtualenvs/careless_navigator/lib64/python3.11/site-packages/torch/onnx/_internal/onnx_proto_utils.py”, line 175, in _export_fileopened_file.write(model_bytes)OSError: [Errno 28] No space left on deviceBTW, this happens with use_merged=False also. Something has changed since last time I ran the same code. Before I was able to get to the optimizer.optimize(…) step without this error showing. I have moved the model location to a 3TB disk with 1.8TB free space, with same results.I verified that the CLI execution still runs correctly and produces an optimized ORT model.Regardless, when I look at the source code for class ORTOptimizer, line 101“if model_or_path.use_merged is True:” would raise raise NotImplementedError.Best,Borell
2023-08-14T17:08:26Z
Hi,I am wondering if this issue is being looked at by anyone, or should it be closed w/o resolution? I will presume that only the CLI version is a workable solution.Best,Borell
2023-08-17T09:27:12Z
Hi Borell, it should not be the case. Can you open an issue on Github with a reproduction?
2023-08-17T14:27:47Z
@fxmarty,Happy to do that. Can you please clarify on which Github to post?
2023-08-17T14:49:36Z
@BorellCool! You can open an issue in the GitHub repo of Optimum:Issues · huggingface/optimum · GitHub
2023-08-17T19:16:58Z
@fxmarty,@regisss,While trying to replicate the issue for posting into GitHub I made a change in the code which seems to have resolved the problem. In the following line,model = ORTModelForCausalLM.from_pretrained(model_path, export=True, use_merged=True)I changed use_merged = False. It was originally set to True as a suggestion to save memory utilization. I don’t fully understand the reason, but both CLI and programmatic executions work identically now.The original model store size was 144.4GB reduced to 22.2GB.I believe this issue can be close.Thanks,Borell