CUDA OOM on model(inputs) but not on model.generate(inputs), but doesn't generate use model(inputs)?
using the same model inputskwargs = {
     "input_ids":input_ids,
     "attention_mask":attention_mask,
 }i get a CUDA out of memory when i dooutputs = model(**kwargs)but not when i dooutput_ids = model.generate(
    **kwargs,
    do_sample=False,
    max_new_tokens=max_output_len,
    pad_token_id=tokenizer.eos_token_id,
    top_p=None,
)doesnt model.generate do amodel(**kwargs)like operation several times? why is my version so memory inefficient?model ismicrosoft/phi-1_5and transformers.version4.40.1
2024-05-04T19:42:36Z
Hi!The reason is because onmodel.forward()you will have gradients being calculated unless you do:with torch.no_grad():
    model.forward(**inputs)Thegeneratemethod already has the no gradient decorator so it does not use much memory
2024-05-04T19:56:59Z
ah of course! thank you raushan! the community delivers
2024-05-04T19:59:19Z
as a follow up tho, wouldmodel.eval()not do the same thing? aswith torch.no_grad():
    outputs = model.forward(**inputs)

print(outputs)
2024-05-04T20:27:20Z
No,model.eval()is used to transform model layers that to eval mode, i.e. batchnorm or dropout layers.Andtorch.no_grad()deactivates gradient calculation, saving memory usage