Unpacking transformer's trainer.eval() to see every example's output, loss
I’m sure this is an FAQ, but I’ve not found the answer anywhere:I have a transformer trained with BertForMultipleChoice model, andtrainer.eval()producessummarystatistic metrics.  I want to getper-exampleresults over the eval_dataset.I’ve tried something like this, invoking the model’s tokenizer on the sample text:for inp in eval_dataset:
    id = inp['example_id']
    lbl = inp['label']
    txt = inp['citing_prompt']
    toks = tokenizer.tokenize(txt)
    outs = model(toks)
    print(id,lbl,outs)but this complains because I’m providing a list of tokens vs what the model expects:outs = model(toks)
       ^^^^^^^^^^^File “…/torch/nn/modules/module.py”, line 1518, in _wrapped_call_implreturn self._call_impl(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File “…/torch/nn/modules/module.py”, line 1527, in _call_implreturn forward_call(*args, **kwargs)^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File “…/transformers/models/bert/modeling_bert.py”, line 1662, in forwardnum_choices = input_ids.shape[1] if input_ids is not None else inputs_embeds.shape[1]what additional work (beyond tokenizing) do I need to reproduce the model’s embedding prior to the encoded input the model expects? I am also interested in capturing each example’s encoding (eg, to look at similarities among them).or, is there a way/place to tap into the transformer’s eval to get the per-example stats I need?thanks for any hints!
2024-04-08T23:57:26Z
The place withintrainer.eval()that come closest to the nexus I was seeking seems to ~transformers.trainer.Trainer.compute_loss()(l#3059).But for now my self-answer seems to be using the captum.ai utilities.
2024-04-09T02:39:23Z
I’m pretty new to NLP so please forgive me if I’m wrong but I think the problem is that you’re not really give the model what it wants. My impression is that the Trainer class expects data in the form of the output oftokenizer(text_data)and data in the form oftokenizer.tokenize(text_data)image1278×150 5.15 KB
2024-04-09T02:46:50Z
The printed results under tokenizer(text) aren’t actually the direct outputs, I converted the output with thedictfunction before printing
2024-04-09T14:47:22Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.