How to output loss from model.generate()?
I need the probability distribution of word generation to calculate the loss in my original loss function.In particular, the model differs from a normal loss function in that it generates sentences in the same way as when testing.Therefore, I have tried to calculate the loss using model.generate(), but this method does not leave me with the calculated graph needed to calculate the gradient. Could this be easily solved by passing a special argument to the method? Or is there an equally simple solution? Or do I have to implement my own function that generates the text in a way that leaves the computed graph?
2022-04-22T01:36:17Z
I can’t answer many of your questions, but I did find this code snippet useful to get a computational graph withgenerate():from undecorated import undecorated
from types import MethodType

generate_with_grad = undecorated(model.generate)
model.generate_with_grad = MethodType(generate_with_grad, model)Thegenerate()function has ano_graddecorator that stops the computational graph being returned, and this code just removes the decorator and leaves the rest of the generate function unchanged.
2022-04-22T05:37:36Z
Thanks for sharing your simple solution!I’ll give this method a try!
2022-04-22T06:03:56Z
thanks!tomroth1001!I got scores with computational graph!
2022-10-01T19:38:05Z
Hello, I tried this method to retain the computational graph and it works. However when I try to backpropagate some loss computed from the generation scores I get an error:RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operationWhich is triggered somewhere in generation_logits_process.py.Last few lines of the trace:next_token_scores_processed = logits_processor(File “/home/halamvac/venvs/venv39/lib/python3.9/site-packages/transformers/generation_logits_process.py”, line 92, incallscores = processor(input_ids, scores)File “/home/halamvac/venvs/venv39/lib/python3.9/site-packages/transformers/generation_logits_process.py”, line 161, incallscore = torch.gather(scores, 1, input_ids)@mittu@tomroth1001Did any of you make it work with backpropagating the gradient?
2022-10-01T23:30:40Z
Not sure, sorry. It worked for my case.The cliché advice is to make sure all your packages are up to date and then try again—might be a bug.
2022-10-03T11:50:05Z
Thanks for the tip but I already have the latest versions. Might I ask what versions of pytorch and transformers are you using?
2022-10-04T19:58:12Z
I’m using transformers 4.19.2 and torch 1.11.0+cu113, and it still work with out errorAnd my minimum code is here.from transformers import BartForConditionalGeneration
from undecorated import undecorated
model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")
generate_with_grad = undecorated(model.generate)
model.generate_with_grad = MethodType(generate_with_grad, model)

output=model.generate_with_grad(
  input_ids=input_ids,
  output_scores=True,return_dict_in_generate=True,
  output_hidden_states = True,
  )According to error message, there is an in-place operation at somewhere, so It would be a good idea to do a backward look, commenting out each calculation process line by line from the end to the beginning, to find which calculation process has the problem. Note backward() is possible only for scalars, so for multidimensional tensors, it is better to sum or average as appropriate.
2024-01-26T04:22:42Z
I have a question.Are there concerns that temporarily removing@torch.no_grad() could lead to learning during the test phase?
2024-04-18T01:01:16Z
I got the following error while trying theun-decorating method.TypeError: _DecoratorContextManager.clone() got an unexpected keyword argument 'input_ids'Package versionstorch == 2.1.0+cu121
transformers == 4.33.2
undecorated == 0.3.0
2024-04-18T18:30:22Z
@jhuangHi Jiaji, I have the same problem as you, did you fix it? Thank you~
2024-04-19T01:16:53Z
Yes, this works for meoutput = model.generate.__wrapped__(model, input_ids, output_scores=True, return_dict_in_generate=True)You should seeoutput.scores[0].requires_grad=True.Note:model.generate.__wrapped__is the actual functiongenerate(), getting rid of the@no_grad()decorator. But as the function is a method of a python class, we’d have to input something for its first argumentself. So here I simply putmodelas first argument.Theundecorate methoddoesn’t work, as it gets the decorator@no_graditself rather than the actualgenerate()function.
2024-04-25T09:24:08Z
Hi Jiaji， Sorry for the late reply. I have been testing according to the method you provided these days, thank you very much!Thanks again for your quick response.
2024-04-25T17:22:11Z
@liyongkangNo problem! However, I do observe some notable difference in gradients against the following approach:generate as usual with no_gradforward pass of prompt concatenated with generated tokens.calculate some loss function and call loss.backward()I am curious about your test outcome. Let me know what you find?
2024-04-25T21:27:35Z
Hi Jiaji, Maybe it is because output.scores are not the raw logits, you cancheck this. And setoutput_logits=Truethen it will return the raw logits.I originally hoped that this gradient would help me have more control over the generated content. However， I found that I couldn’t even control the length of the generated text (I can only specify the maximum length, but not the generated text to a certain length. If the length is uncertain, I cannot calculate my loss. ), so I switched to other strategies.Thanks again for your help.
2024-04-26T17:29:07Z
I’m pretty sure in both cases, they are raw logits. In fact, the difference between the logits in these two cases are tiny. But the backward() results in notable difference in gradient. I’d say the steps 1-3 list above is a safer choice.