Adapting replit transformers support for training
Hello,I am trying to finetune replit code model (replit/replit-code-v1-3b · Hugging Face) using the same procedure in alpaca-lora, aiming to make the model instruction following.However, when I try to use the model with transformers.Trainer, it turns out the replit model forward function does not supportlabelsand input_embeds which I guess are used by Trainer to update the weights:ReplitLM/replit_lm.py at 1a421995c10c880379f52e26b5ba495c87bac3e2 · replit/ReplitLM · GitHubI’m wondering if there’s a guide or specific steps I need to follow in order to modify the replit model code base so that it supports training with Trainer class