Domain adaptation fine tune VS instruction_tuned
TL;DR: I want to fine-tune ‘meta-llama/Llama-2-7b-chat-hf’ with my own instructional data, but it turns out that the new model is worse than the original one.How should I differentiate in the code between different types of fine-tuning? I saw different types of fine-tuning on the internet:Domain adaptionInstruction-based fine-tuning (it looks like this is what is needed)Supervised fine-tuning …But how should it be different in the code? For example, assuming that I use AutoModelForCausalLM and a trainer object, a simple code would look like this:model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
    trainer = Trainer(

        model=model,
        args=training_args,
        train_dataset=dataset_train,
        eval_dataset=dataset_test,
        data_collator=transformers.DataCollatorForLanguageModeling(
            tokenizer, mlm=False
        )But how dataset_train should look like ?
2024-01-19T10:06:52Z
Hi, I’ve you found the answer at your question?I’m interested by your experience.
2024-01-21T13:20:16Z
Hicfrancois7,I have discovered the answer to your question.Essentially, whattrainer.train()does is attempt to predict the next token. Therefore, when you provide to the trainer atrain_datasetoreval_datasetthat consists of a list with a token or string in each element, the goal is invariably to predict the subsequent token.From this perspective, there is no distinction between Domain Adaptation and Instruction-based Fine-tuning – the difference lies solely in the data.In Domain Adaptation, the data usually takes the form of a single, very long text that has been concatenated. The objective remains next-token prediction, where the context is provided in windows that typically have a specified context length parameter (e.g., 2K, 4K, …). This stage often encompasses the bulk of the learning due to the extensive size of the data.When it comes to Instruction Tuning, it usually involves a few thousand pairs of prompts and completions aimed at teaching the model how to respond to instructions. However, generally, there is no difference in the learning method – the model always attempts to predict the next token, whether it is part of the prompt or the completion.Please note that you can optionally configure the model to learn only from the completion part by using theSFTTrainerfrom TRL instead of the regularTrainer. Essentially,SFTTraineris designed to overrideTrainer.For more information, seeTrain on Completions Only.