How to set up Trainer for a regression?
Hello,I am aware that I can run a regression model usingfloattarget values andnum_labels=1in a classification head like below:model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english", 
                                                           num_labels=1,
                                                           ignore_mismatched_sizes=True)The problem is that right now I am merely adapting theTrainerspecs for classification and during training I can see anaccuracymetric wherermseorr-squaredwould be more appropriate.See theaccuracyscore below on the validation data:from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

batch_size = 32

args = TrainingArguments(
    evaluation_strategy = "epoch",
    save_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=3,
    report_to="none",
    weight_decay=0.01,
     output_dir='/content/drive/MyDrive/kaggle/',
    metric_for_best_model='accuracy')

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['test'],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)which givesEpochTraining LossValidation LossAccuracy10.5073000.4996250.50385320.4660000.4957240.503853Which arguments intrainershould I use to I getrmseorr-squaredinstead?I assume the loss (that is minimized) is already the mean squared error (maybe I am wrong?)Thanks!
2021-12-21T13:44:53Z
You can use the RMSE metric, like so:from sklearn.metrics import mean_squared_error

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    rmse = mean_squared_error(labels, predictions, squared=False)
    return {"rmse": rmse}Sources:scikit learn - Is there a library function for Root mean square error (RMSE) in python? - Stack Overflow
2021-12-21T13:58:25Z
ahhh!! thanks@nielsr!! and am I correct to assume that the loss being minimized here (0.5 and 0.466 in the example) is the mean squared error as well?
2021-12-21T13:59:29Z
Yes, you can see that in the source codehere.Note that you can also set theproblem_typeof the model to “regression” (which is equivalent to settingnum_labels=1).
2022-11-21T16:53:57Z
Using HF Evaluatefrom evaluate import load
metric = load('mse')

metric.compute(predictions=predictions, references=labels, squared=False)
2023-06-28T11:37:56Z
@nielsr, can you please help me with which language model I can use here? For example I am solving a rating prediction task for user-movie in the range of [1-5]. But when I am doing num_labels =1, it says IndexError: Target 4 is out of bounds. Can you pls help me?
2024-04-13T19:51:24Z
Hi@olaffson! I’m trying to use AST for sound regression (for college) and I’m struggling on how to train a model based on my own data. Do you have any reference on how to do it or can I find your code in any public repository? I appreciate any help