Returning Multiple Answers for a QA Model on SageMaker
Hi,I’ve currently fine-tunedthisQuestion-Answering model to fit a specific business use case we have (identifying the name of a company from a piece of text). When it comes to inference, I’ve found as@sguggerhas very clearly explained inthis notebookthat sometimes the best answer isn’t the one with the best start and end logits as sometimes the highest scoring combination can produce an answer that is too long or too short (just one character).As such when I was predicting using this model locally I created a return_best_combinaton function that finds the most practical answer using the list of logit scores.When I used this model using the SageMaker API I realised it just returns one single answer with a score assigned to it. I wanted to check how this answer is produced? (happy to just be directed to the source code if it’s available) and wether it’s possible to return n number of likely answers instead of just 1.Thanks every so much,Karim
2021-11-18T09:32:51Z
The Inference Toolkit uses the transformerspipelinesunder the hood. So if your are deploying a model for Question-Answering it would use thepipeline("question-answering"). You can find the code for this here:transformers/question_answering.py at main · huggingface/transformers · GitHubBut if you want to use your own functionreturn_best_combinatonyou could create a custominference.pywith your own “prediction” step.huggingface.coDeploy models to Amazon SageMakerWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2021-11-18T15:17:09Z
Understood that’s very helpful thank you. Looking at the code it seems thatmax_answer_len,handle_impossible_answer&topkall get me what I’m looking for so that’s perfect! No need for my owninference.pyscript. Thank you!
2023-01-14T12:15:06Z
Can you please elaborate how you fixed it? Would be really helpful as I am dealing with the same problem right now.This is the model I am trying to get the multiple inferences from, for which Sagemaker is returning just 1.hub = {‘HF_MODEL_ID’:‘valhalla/t5-base-qa-qg-hl’,‘HF_TASK’:‘text2text-generation’}
2023-01-23T06:59:42Z
Hey@casafurix! You can use thetop_kparameter to get more than one answer. If you set it to5for example you’ll get the top 5 answers ranked by scores.