Preventing every dropout in the GPT2DoubleHeadsModel
Hello,If I execute the lines below, would they prevent every dropout in theGPT2DoubleHeadsModel? Or do I need to change extra settings in order to completely prevent the dropout in myGPT2DoubleHeadsModel? Thank you,# disable dropouts for the model_gpt2_double_heads.
    model_gpt2_double_heads.config.resid_pdrop = 0.
    model_gpt2_double_heads.config.embd_pdrop = 0.
    model_gpt2_double_heads.config.attn_pdrop = 0.
    model_gpt2_double_heads.config.summary_first_dropout = 0.
2020-09-30T00:34:00Z
You should do that when instantiating your model:GPT2DoubleHeadsModel.from_pretrainec(checkpoint, resid_pdrop=0, embd_pdrop=0., ...)for instance
2020-09-30T01:18:29Z
Thank you for your reply.Is there more dropouts forGPT2DoubleHeadsModelother than theresid_pdrop,embed_pdrop,attn_pdrop, andsummary_first_dropout?Thanks again,
2020-09-30T13:57:08Z
Those seem to be the only ones (we can see all parameters in the config file of GPT-2here).
2023-06-07T07:51:47Z
Hi@h56choand@sgugger,I hope you are well. sorry, I am doing gpt-neo fine tunning. for controlling the overfitting I want to increase drop out in the model to -.2 . can I do it as recommended above? after that  can I use the model for fine tunning as a pretrained model? or it needs training from scratch? many thanks