Get the positive score in a classification task by using a generative model
Hi, I’m attempting to utilize a generative model (Llama2) for a binary classification task and aim to obtain the positive score, which represents the confidence level for the positive label.I tried to usecompute_transition_scoresbut not sure how can I get the confidence between 0-1 correctly.Here is my current code:model = AutoModelForCausalLM.from_pretrained(
    peft_config.base_model_name_or_path,
    # quantization_config=bnb_config,
    torch_dtype='auto',
    device_map='auto',
    offload_folder="offload", offload_state_dict = True
)
pos_scores = []
input_ids = tokenizer(test_sample, return_tensors="pt").input_ids
tokens_for_summary = 1
output_tokens = input_ids.shape[1] + tokens_for_summary

outputs = model.generate(inputs=input_ids, do_sample=False, max_length=output_tokens, pad_token_id=tokenizer.eos_token_id, 
                         output_scores=True, return_dict_in_generate=True)
score = float(torch.exp(model.compute_transition_scores(outputs.sequences, outputs.scores)).cpu())
        
if pred_label == 1:
   pos_scores.append(score)
elif pred_label == 0:
   pos_scores.append(-1 * score) # reverse the sign of all samples for which the prediction was 0.However, I’m obtaining high values. I’ve considered using the sigmoid function, but I’m not entirely certain if this is the correct approach.How should I do that? Thank you!