Negative Kl values during PPO training (TRL library)
Hey everyone!I am trying to train my model using PPO trainer from TRL library. However I receive negative kl values. any Idea where might have gone wrong?The configs:generation_kwargs = {“do_sample”:True,“top_k”:9,“max_length”:1024,“top_p”:0.9,}dataset = train_datasetppo_config = {“mini_batch_size”: 1,“batch_size”: 1,“learning_rate”: 1.41e-5,}ppo_trainer = PPOTrainer(config, model, tokenizer = tokenizer, dataset = dataset)