Eval with trainer not running with PEFT LoRA model
Hi there,I have a training script that when running with defined model and trainer, the evaluation loop works fine.When wrapping the model with PEFT LoRA config evaluation doesn’t run. The training is completed properly but the evaluation loop every X steps just doesn’t seem to happen.When I dug in a bit I saw it happened because for some reason in PEFT mode the trainer didn’t provide labels to the evaluation loop so it was skipped.Here’s the package versions and script:torch==2.0.1
transformers==4.32.1
peft==0.5.0Original code:class CLIPForSceneClassification(CLIPModel):

    def forward(self,
        pixel_values,
        labels,
        text_inputs):

        s = pixel_values.shape # B,S,3,H,W
        pixel_values = pixel_values.view([s[0]*s[1],s[2],s[3],s[4]])
        outputs = self.vision_model(pixel_values=pixel_values)
        outputs = outputs[1].view([s[0],s[1],-1])
        image_embeds = torch.mean(outputs,dim=1)

        text_outputs = self.text_model(**text_inputs)
        text_embeds = text_outputs[1]

        image_embeds = self.visual_projection(image_embeds)
        text_embeds = self.text_projection(text_embeds)

        # normalized features
        image_embeds = image_embeds / image_embeds.norm(p=2, dim=-1, keepdim=True)
        text_embeds = text_embeds / text_embeds.norm(p=2, dim=-1, keepdim=True)

        # cosine similarity as logits
        logit_scale = self.logit_scale.exp()
        logits = torch.matmul(image_embeds, text_embeds.t()) * logit_scale
        # logits_per_text = logits_per_image.t()

        loss = None
        if labels is not None:
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, self.n_classes), labels.view(-1))

        return loss, logits

def compute_metrics(pred):
        labels = pred.label_ids
        preds = pred.predictions
        preds = preds.argmax(-1)
        precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average="weighted")
        acc = accuracy_score(labels, preds)
        return {"accuracy": acc, "f1": f1, "precision": precision, "recall": recall}

model = CLIPForSceneClassification.from_pretrained(args.model_name)

training_args = TrainingArguments(
    output_dir=args.model_dir,
    num_train_epochs=args.epochs,
    per_device_train_batch_size=args.train_batch_size,
    per_device_eval_batch_size=args.eval_batch_size,
    warmup_steps=args.warmup_steps,
    evaluation_strategy="steps",
    eval_steps=50,
    logging_dir=f"{args.output_data_dir}/logs",
    learning_rate=float(args.learning_rate),
    # label_names=["real", "animated"],
    report_to="none",
    use_mps_device=torch.backends.mps.is_available(),
    logging_steps=10,
    save_strategy="steps",
    save_steps=50,
    dataloader_num_workers=2,
    remove_unused_columns=False,
)

# create Trainer instance
trainer = Trainer(
    model=lora_model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
)Here’s what I modified with PEFT:config = LoraConfig(
    r=4,
    lora_alpha=4,
    lora_dropout=0.0,
    target_modules=["visual_projection", "text_projection"],
    # modules_to_save=["seq.4"],
)
lora_model = get_peft_model(model, config)
lora_model.print_trainable_parameters()

...

# create Trainer instance
trainer = Trainer(
    model=lora_model,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_dataset,
    eval_dataset=test_dataset,
)Would be happy for help here. Thanks!!!
2023-09-10T04:55:02Z
They haven’t made PEFT compatible with thefind_labelsfunction yet.The easiest fix should be to add this argument in training args.from transformers import TrainingArguments
training_args = TrainingArguments(...,label_names=['labels'])