How to find the beam search score for any target output? (BartForConditionalGeneration)
Hi,I’m writing a function that calculateslog[\mathbb{P}_{model}(target\mid input)], which is thesequences_scoresin the beam search output, for anyinputandtarget. I tried in the following ways:Adapt the codes ofgreedy_search()so that at each step it takes a token oftargetasnext_tokens. The result is the sum of log probs fromoutput.scoresdivided bylen(target)^{length\_penalty}.Adapt the codes ofbeam_search()to accpetnum_beams=1, then uses thetargettokens asbeam_next_tokensat each step.But when I used the sequences returned bygenerate()for a sanity check, I got differences ofO(10^{-3})in the results for both methods. Also, I noticed that the scores of beam search are not “deterministic” and are influenced by the results of other beams. As shown in the figure, 9/10 of the beams find the same sequence but yields different scores.Screenshot 2022-03-21 at 15.56.152204×547 98.4 KBI would like to know what might be the subtleties in the beam search calculation? I’ve also read the codes ofBeamSearchScorerandBeamHypothesisbut couldn’t figure out the reason. Or, is there a correct way for the probability estimation?Moreover, besides the normalisation bylen(target)^{length\_penalty}, what are the other factors that can produce a mismatch of a sequence’s probabilities inbeam_searchandgreedy_search?Thanks in advance for your help.