text
Hey GPT, check yourself...
Hey GPT, check yourself...
new_line
new_line
text
Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥
Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥
new_line
new_line
text
💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.
💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.
new_line
new_line
text
This method is called SelfCheckGPT with Prompt and shows very nice results. 👀 
This method is called SelfCheckGPT with Prompt and shows very nice results. 👀 
new_line
new_line
text
The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃
The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃
new_line
new_line
text
More details and variations of this method are in the paper: 
More details and variations of this method are in the paper: 
resource
https://huggingface.co/papers/2308.00436
paper
2308.00436
https://huggingface.co/papers/2308.00436
SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step
  Reasoning (2308.00436)
Hey GPT, check yourself...

Here is a black-box method for hallucination detection that shows strong correlation with human annotations. 🔥

💡 The idea is the following: ask GPT, or any other powerful LLM, to sample multiple answers for the same prompt, and then ask it if these answers align with the statements in the original output. Make it say yes/no and measure the frequency with which the generated samples support the original statements.

This method is called SelfCheckGPT with Prompt and shows very nice results. 👀 

The downside, we have to do many LLM calls just to evaluate a single generated paragraph... 🙃

More details and variations of this method are in the paper: https://huggingface.co/papers/2308.00436
👍
arora-anmol osanseviero clem samusenps victor pulkitmehtawork Gvangelista yjernite
❤️
clem samusenps