Difference between CausalLMWithValueHead vs ModelForCausalLM
I know what isAutoModelForCausalLM. The thing I’m asking is that in thepeftLoRA Fine tuning tutorial, the autors have usedAutoModelForCausalLMWithValueHeadwhile you pick any code or notebook on Fine-tuning of any LLM withPEFTstyle, you’ll findAutoModelForCausalLMbeing used.I went to lean on theofficial documentation ofAutoModelForCausalLMWithValueHeadand found:An autoregressive model with a value head in addition to the language model headWhat I want to ask is thatHow, where and more importantly,WHYthis extraValueHeadis used
2024-01-26T21:27:18Z
While looking for an answer to this question, I came across this discussion. Is it helpful?what is AutoModelForCausalLMWithValueHead? · Issue #180 · huggingface/trl · GitHub
2024-02-15T09:23:47Z
First things first, this additional ValueHead has nothing to do with PEFT.Mainly, PPO optimization (a RLHF technique) relies on computing “advantages” associated with taking a particular action (in this case, selecting a token) in a particular state. The computation relies on value of (state,action) pair minus, the value of being in the state. You can review the exact calculation in this function:trl/trl/trainer/ppo_trainer.py at main · huggingface/trl · GitHubThe additional ValueHead simply projects the last hidden states onto a scalar to estimate the value of a state. Check the ValueHead class implementation here:trl/trl/models/modeling_value_head.py at main · huggingface/trl · GitHubNote: The ValueHead class is only needed if you plan to perform training/re-training.