Fine-tuning GPT2 Family (Small to XL), How should hyperparameters and generation criteria change?
Hello everyone,I’m working on a project that includes finetuning the entire GPT2 family (from small to XL) on ~3000 prompt completion pairs of short-ish length (around 150 characters each).So far, with the same hyperparameters, I am observing much more coherent language generation for my fine-tuning task with GPT2 small in comparison to GPT2XL. Also viewing train loss and eval loss gives no indication of overfitting. I figure that this has to be a function of my training (or sampling) process. Could anybody give me some intuition on why a larger model might perform worse with the same hyperparameters and how I should vary hyperparameters, training process or generation criteria as I increase model size?My hyperparameters:epochs: 7 (but with early stopping)early_stopping_patience: 10 (on eval_loss)bf16: truelearning_rate: 0.0003epsilon (1e-7) for AdamWbatch_size: 1 (eval and train)weight_decay: 0.01My sampling/generation criteria:max_length: 300temperature: 0.6repetition_penalty: 1.1top_k: 100top_p: 1My intuition here is epochs play a major role. Would it be fair to assume a smaller model needs more instruction (and thus more epochs?). I think that my larger model (GPT2XL) might only need 2 or 3 epochs since it has better language generation from its original pre-trained self than GPT2-small. That being said the train/eval curves do not tell that story.Thanks in advance!Andrew