Regarding Quantizing gpt2-xl, gpt2-large, &c
Hello,I have spent some time now trying to quantizegpt2-largeand am having trouble. I’m following the example providedhere. The process fails when runningquantizer.exportThe specific error is:FileNotFoundError: [Errno 2] No such file or directory: '.../transformer.wte.weight'The error is self-explanatory. Judging bythis discussionand theuse_external_data_formatflag (given the model is >2GiB), I am meant to be storing the gpt2 weights intransformer.wte.weight,etc., but I cannot, for the life of me, figure out how to export these files withORTModelForCausalLM.save_pretrained.Should I be picklingmodel.transformer.wte.weight,model.transformer.wpe.weight,etc. manually? I’ve tried loading the model with AutoModel and dumping the required files but no dice; mainly because I can’t dump model weights liketransformer.h.0.ln_1.weight.Any help would be much appreciated.
2022-08-10T08:02:12Z
Here you go, I’ve prepared a gist here:Converting gpt2-large to onnx with multiple external files and using it later for inference · GitHubOne script will create a folder with the exported .onnx and all of its depended external files as well as a copy of the tokenizer. The other script will load the local onnx folder and use it for inference.I’d be happy to know if you manage to quantize it and/or manage to store it in the hub in a way which doesn’t break the model pageUpdate: You might also want to check out this repo:GitHub - ELS-RD/transformer-deploy: Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀
2022-08-10T21:52:23Z
Thank you kindly for both the gist and the repo! It had really been nagging me. Happy to report back that all is well and works as expected. Cheers.