text
🔎 DPO hyperparameter search update!
🔎 DPO hyperparameter search update!
new_line
new_line
text
In my previous post (
In my previous post (
link
https://huggingface.co/posts/BramVanroy/633544255876795
https://huggingface.co/posts/BramVanroy/633544255876795
text
), I indicated how despite high reward accuracies and low losses, my model would sometimes just output repeating random tokens (
), I indicated how despite high reward accuracies and low losses, my model would sometimes just output repeating random tokens (
inline_code
`/*****/`
/*****/
text
). There were some useful brainstorms in that thread. I think the dataset is relatively easy for the model, leading it to quickly overfit when the beta is very small, which allows the model to step away further from its initially outputs.
). There were some useful brainstorms in that thread. I think the dataset is relatively easy for the model, leading it to quickly overfit when the beta is very small, which allows the model to step away further from its initially outputs.
new_line
new_line
text
So, I ran a hyperparameter search for learning rate (1e-7 v 5e-7), batch size (32, 64, 96, 128) and most importantly, beta (0.01, 0.1, 0.2, 0.5). You can have a look at the results for yourself here: 
So, I ran a hyperparameter search for learning rate (1e-7 v 5e-7), batch size (32, 64, 96, 128) and most importantly, beta (0.01, 0.1, 0.2, 0.5). You can have a look at the results for yourself here: 
link
https://wandb.ai/bramvanroy/dpo-geitje-ultra-hyperparams
https://wandb.ai/bramvanroy/dpo-geitje-ultra-hyperparams
new_line
new_line
text
Interpreting the result, I'd think that the beta=0.5 is the better choice for this dataset. Reasons:
Interpreting the result, I'd think that the beta=0.5 is the better choice for this dataset. Reasons:
new_line
new_line
text
- markedly higher rewards margins compared to all other betas
- markedly higher rewards margins compared to all other betas
new_line
text
- better balance between positive chosen and negative rejected rewards
- better balance between positive chosen and negative rejected rewards
new_line
text
- log probabilities are not as superbly low as for beta=0.01, which seems too low for this dataset
- log probabilities are not as superbly low as for beta=0.01, which seems too low for this dataset
new_line
new_line
text
Of course, that is just purely looking at numbers without running any benchmarks. However, I am hesitant to evaluate all the models on benchmarks and, therefore, literally optimising my hyperparameters on a test set (which is very bad!). So I will just play with some of the most promising models and see which one feels "best" qualitatively.
Of course, that is just purely looking at numbers without running any benchmarks. However, I am hesitant to evaluate all the models on benchmarks and, therefore, literally optimising my hyperparameters on a test set (which is very bad!). So I will just play with some of the most promising models and see which one feels "best" qualitatively.
new_line
new_line
text
If you have other insights, thoughts, or opinions, let me know!
If you have other insights, thoughts, or opinions, let me know!
🔎 DPO hyperparameter search update!

In my previous post (https://huggingface.co/posts/BramVanroy/633544255876795), I indicated how despite high reward accuracies and low losses, my model would sometimes just output repeating random tokens (`/*****/`). There were some useful brainstorms in that thread. I think the dataset is relatively easy for the model, leading it to quickly overfit when the beta is very small, which allows the model to step away further from its initially outputs.

So, I ran a hyperparameter search for learning rate (1e-7 v 5e-7), batch size (32, 64, 96, 128) and most importantly, beta (0.01, 0.1, 0.2, 0.5). You can have a look at the results for yourself here: https://wandb.ai/bramvanroy/dpo-geitje-ultra-hyperparams

Interpreting the result, I'd think that the beta=0.5 is the better choice for this dataset. Reasons:

- markedly higher rewards margins compared to all other betas
- better balance between positive chosen and negative rejected rewards
- log probabilities are not as superbly low as for beta=0.01, which seems too low for this dataset

Of course, that is just purely looking at numbers without running any benchmarks. However, I am hesitant to evaluate all the models on benchmarks and, therefore, literally optimising my hyperparameters on a test set (which is very bad!). So I will just play with some of the most promising models and see which one feels "best" qualitatively.

If you have other insights, thoughts, or opinions, let me know!
👍
osanseviero tomaarsen kashif edbeeching davanstrien philschmid dingo-actual Rijgersberg dvilasuero gabrielmbmb abidlabs julien-c HumanFace Cherran
https://cdn-avatars.huggingface.co/v1/production/uploads/1627505688463-60107b385ac3e86b3ea4fc34.jpeg
Daniel van Strien
davanstrien
user
https://cdn-avatars.huggingface.co/v1/production/uploads/1594192845975-5e1e17b6fcf41d740b6996a8.jpeg
Bram Vanroy
BramVanroy
user