When exporting seq2seq models with ONNX, why do we need both decoder_with_past_model.onnx and decoder_model.onnx?
Hi there, I’m the creator ofTransformers.js, a JavaScript library which aims to run HuggingFace models directly in the browser. It relies on optimum to convert PyTorch models to ONNX, which can then be used inside web browsers usingonnxruntime-web. For the most part, everything is working fine, but there appears to be a ton of duplicate parameters between decoder_with_past_model.onnx and decoder_model.onnx, for sequence to sequence and causal language models.For causal language models, such as GPT2, I was able to avoid having to use decoder_model.onnx by creating an empty tensor for past_key_values, and passing the inputs through the decoder_model_with_past.onnx (seehere).I tried employing the same trick for sequence to sequence models (e.g., T5, Bart and Whisper), but ran into a problem since they need past key values for the encoder too. Once again, I tried just creating dummy inputs with a dimension of 0 where concatenation would take place. However, this didn’t work either, as the outputted “present” key value pairs were all empty tensors too.I asked on the Discord channel, and I was directed to the forum to see if anyone could provide some assistance. Perhaps being able to export smaller parts of the models to avoid significant parameter duplication? This would dramatically improve performance (cutting down model size by up to 50%)!I look forward to any responses! Thanks~ Xenova
2023-03-08T20:31:18Z
Hi@Xenova, thank you for having a try at this!I have the same experience as you for gpt2 (although I had to addposition_idsas inputs to have matching logits due tothis logic) using onlydecoder_with_past_model.onnx.Unfortunately I did not have time to try with encoder-decoder models, I was assuming it was possible. I could have a look shortly.Alternatively, are you able to support ONNX models that have subgraphs? That’s the approach we are currently taking in Optimum, for reference:Validating ONNX model fails for GPT-J · Issue #607 · huggingface/optimum · GitHub. This is currently available only for decoder-only models for now though, I plan to extend to encoder-decoder architectures.If you export a decoder-only model you’ll see as output a merged decoder in charge of both cases without past/with past:optimum-cli export onnx gpt2 gpt2_onnx/
2023-03-08T20:41:41Z
I have the same experience as you for gpt2 (although I had to addposition_idsas inputs to have matching logits due tothis logic) using onlydecoder_with_past_model.onnx.Are you still having the problem, even if you pass an empty tensor (of shape [batch_size, this.num_heads, 0, this.dim_kv])? I was able to bypass that check since, in that case, it isn’t None, and then the concatenation logic works.As you can see, here is the code to create the model (only uses decoder_with_past_model.onnx):github.comxenova/transformers.js/blob/c2e47f61396749a77d767d2572e440fd3c5db8b5/src/models.js#L145-L171class AutoModelForCausalLM {static async from_pretrained(modelPath, progressCallback = null) {// , decoder_with_past_model// let name = use_past ?  : 'decoder_model.onnx'let [config, session] = await Promise.all([fetchJSON(modelPath, 'config.json', progressCallback),constructSession(modelPath, 'decoder_with_past_model.onnx', progressCallback)])// Called when all parts are loadeddispatchCallback(progressCallback, {status: 'loaded',name: modelPath});switch (config.model_type) {case 'gpt2':return new GPT2LMHeadModel(config,This file has been truncated.show originaland it works correctly (see thedemofor proof).Unfortunately I did not have time to try with encoder-decoder models, I was assuming it was possible. I could have a look shortly.That would be greatly appreciated! I’ve been banging my head against the wall for 2 days now, but I am getting strange outputs (sometimes the logits are NaN, other times the generated present_key_values are empty tensors).Alternatively, are you able to support ONNX models that have subgraphs? That’s the approach we are currently taking in Optimum, for referenceI should say that I am not an expert in ONNX haha; so, I’m not too sure what subgraphs areFor me, at least, I treat ONNX as a magical black box, and I provide the inputs and process the outputs.If you export a decoder-only model you’ll see as output a merged decoder in charge of both cases without past/with pastOkay great! Yes, this is something I desperately need for Transformers.js … the smaller the model is, the better!
2023-03-08T20:54:48Z
Oh, I see, so thatpast_key_values[0][0].size(-2)is zero. I did not have a try at it - did not expect ONNX Runtime to work with zero-sized inputs, but I’ll have a try!Subgraphs is basically a fancy way to handle controlflows (if/else, loops) with ONNX. Depending on whether you need to use k/v cache or not, you can dispatch the compute on one or the other of the two branches of anIfnode, that shares weights between both:image1742×752 59.3 KBAs a rule of thumb though, if you are able to avoid subgraphs (e.g. thanks to the hack with fake k/v), I would recommend to avoid it.
2023-03-08T20:58:38Z
fxmarty:Oh, I see, so thatpast_key_values[0][0].size(-2)is zero. I did not have a try at it - did not expect ONNX Runtime to work with zero-sized inputs, but I’ll have a try!Haha yes, I was quite surprised myself when I got it working with gpt2.As a rule of thumb though, if you are able to avoid subgraphs (e.g. thanks to the hack with fake k/v), I would recommend to avoid it.Understood. Is there anything I can do to help try get this working? I could continue trying myself to get the empty tensor working for encoder-decoder models. (Although, I do think fixingthis issuewill either fix it, or help get closer to our goal!)
2023-03-08T21:11:35Z
If you think there are inputs/outputs that are ill-defined, you could have a look atoptimum/optimum/exporters/onnx at main · huggingface/optimum · GitHub(in order of hierarchy:base.py,config.py,model_configs.pyas explained inhere, and hack and try from there, for exampleoptimum/base.py at 913f9d5fe619acc852210f5978c62e1271185287 · huggingface/optimum · GitHubandoptimum/config.py at 913f9d5fe619acc852210f5978c62e1271185287 · huggingface/optimum · GitHub.I’ll have a look if there’s an issue with the export itself (like whisper it seems), or if it’s fully an implementation problem.
2023-03-09T13:15:08Z
HUGE update! I just got whisper seq2seq working WITHOUT having to load both decoders. This is done using the merged decoder.There were some hacks still necessary though, namely:Creating empty dummy past key values for encoder and decoder when using cache_branchlet encoder_heads = this.config.encoder_attention_heads;
let encoder_dims = [1, encoder_heads, 0, this.config.d_model / encoder_heads];
for (let i = 0; i < this.config.encoder_layers; ++i) {
    decoderFeeds[`past_key_values.${i}.encoder.key`] = new Tensor('float32', [], encoder_dims)
    decoderFeeds[`past_key_values.${i}.encoder.value`] = new Tensor('float32', [], encoder_dims)
}

let decoder_heads = this.config.decoder_attention_heads;
let decoder_dims = [1, decoder_heads, 0, this.config.d_model / decoder_heads];
for (let i = 0; i < this.config.decoder_layers; ++i) {
    decoderFeeds[`past_key_values.${i}.decoder.key`] = new Tensor('float32', [], decoder_dims)
    decoderFeeds[`past_key_values.${i}.decoder.value`] = new Tensor('float32', [], decoder_dims)
}I don’t know if onnx supports simple booleans, so I just created a tensor with a single boolean:use_cache_branch: new Tensor('bool', [pastKeyValues !== null], [1]). Perhaps the if logic checks for all values to be true, and so, it works. In other words, I’m not questioning it too much hahaThere were also a ton of warnings produces, but, it works… so, I don’t really care:image1737×734 117 KBI will see if it is as simple to convert the other models types.
2023-03-10T17:07:32Z
I’m facing the exact problem with whisper model. Could you please specify where you have applied your hacking codes in the conversion?
2023-03-12T03:37:32Z
Here’s a link to the source code:transformers.js/models.js at main · xenova/transformers.js · GitHubHopefully that will be enough for you to decipher!
2023-06-20T16:10:32Z
@Xenovawe are trying to make an App using Quantized whisper-finetuned model and run it in the browser using react our problem that we don’t know how to generate predicted output text from last_hidden_state and logitsGitHubGitHub - iMegz/onnxruntime-web-whisperContribute to iMegz/onnxruntime-web-whisper development by creating an account on GitHub.
2024-01-10T14:48:38Z
Hi@Noahloghman! It seems it couldn’t find the model. What is the model you were trying to load?
2024-03-07T05:15:11Z
Hi@regissssorry for the late reply, its Llama 27B from hugging face.Thank you
2024-03-07T05:19:30Z
Hi guys Hi@xenovaI have this issue :[E:onnxruntime:, inference_session.cc:1533 onnxruntime::InferenceSession::Initialize::<lambda_9a5ue43270b854edk3er320c0a5c4y9a>::operator ()] Exception during initialization: D:\a_work\1\s\onnxruntime\core\optimizer\initializer.cc:31 onnxruntime::Initializer::Initializer !model_path.IsEmpty() was false. model_path must not be empty. Ensure that a path is provided when the modelis created or loaded.I think this still an issue