text
🕵️ Looking for DPO experts!
🕵️ Looking for DPO experts!
new_line
new_line
text
I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of 
I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of 
inline_code
`/*****/`
/*****/
text
. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?
. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?
new_line
new_line
text
Any thoughts? Any suggestions to get around this? All discussions are welcome!
Any thoughts? Any suggestions to get around this? All discussions are welcome!
🕵️ Looking for DPO experts!

I have a dataset that is gpt-4-turbo as chosen and a lower performing model as rejected. The objective should therefore be fairly easy because the two are easy to discern. As a consequence, the model achieves very low losses (0.021 train; 0.013 validation) and high reward accuracies (0.995). **However**, when using the model in practice, it often detoriates after the first one or two tokens and continuously outputs sequences of `/*****/`. So despite the good performance on the DPO objective and strong scores on the validation set (no overfitting), something seems to go wrong. Perhaps the outputs are too different and the task is too easy, in which case DPO is not useful. But why then would the model start hallucinating and repeating the same token over and over again?

Any thoughts? Any suggestions to get around this? All discussions are welcome!
🤗
mrfakename davanstrien clem osanseviero lewtun Dlbk lilsyoss gsarti julien-c clefourrier lunarflu larryvrh
https://cdn-avatars.huggingface.co/v1/production/uploads/1594192845975-5e1e17b6fcf41d740b6996a8.jpeg
Bram Vanroy
BramVanroy
user
/avatars/403b8f796fa7a149b131345c93117af4.svg
Robin Smits
robinsmits
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1647276617786-5f04e8865d08220171a0ad3f.png
François Remy
FremyCompany
user
/avatars/245ec3b183919c079f8c5023b3f7ca9f.svg
CultriX
CultriX
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1594651707950-noauth.jpeg
Lewis Tunstall
lewtun
user
/avatars/ce2c27ba75e4a8004e8616a78d5ff44d.svg
LE WANG
wlhgtc
user
https://cdn-avatars.huggingface.co/v1/production/uploads/60420dccc15e823a685f2b03/Dn7QTyy9SZ7jKN6xpufVD.png
Daniel Vila
dvilasuero
user
https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/JtUGAwVh_4cDEsjNcfpye.png
Maxime Labonne
mlabonne
user
https://cdn-avatars.huggingface.co/v1/production/uploads/618b9a79ba796dc2bf3f4412/66ustwEp1EDU_rxcXJBIO.jpeg
Bram van Es
UMCU
user