text
🔍 Today's pick in Interpretability & Analysis of LMs: A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains by 
🔍 Today's pick in Interpretability & Analysis of LMs: A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains by 
mention
@alonjacovi
alonjacovi
text
mention
@yonatanbitton
yonatanbitton
text
 B. Bohnet J. Herzig 
 B. Bohnet J. Herzig 
mention
@orhonovic
orhonovic
text
 M. Tseng M. Collins 
 M. Tseng M. Collins 
mention
@roeeaharoni
roeeaharoni
text
mention
@mega
mega
new_line
new_line
text
This work introduces a new methodology for human verification of reasoning chains and adopts it to annotate a dataset of chain-of-thought reasoning chains produced by 3 LMs. The annotated dataset, REVEAL, can be used to benchmark automatic verifiers of reasoning in LMs.
This work introduces a new methodology for human verification of reasoning chains and adopts it to annotate a dataset of chain-of-thought reasoning chains produced by 3 LMs. The annotated dataset, REVEAL, can be used to benchmark automatic verifiers of reasoning in LMs.
new_line
new_line
text
In their analysis, the authors find that LM-produced CoTs generally contain faulty steps, often leading to incorrect automatic verification. In particular, CoT-generating LMs are found to produce non-attributable reasoning steps often, and reasoning verifiers generally struggle to verify logical correctness.
In their analysis, the authors find that LM-produced CoTs generally contain faulty steps, often leading to incorrect automatic verification. In particular, CoT-generating LMs are found to produce non-attributable reasoning steps often, and reasoning verifiers generally struggle to verify logical correctness.
new_line
new_line
text
📄 Paper: 
📄 Paper: 
resource
https://huggingface.co/papers/2402.00559
paper
2402.00559
https://huggingface.co/papers/2402.00559
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for
  Verifiers of Reasoning Chains (2402.00559)
new_line
text
🔡 Dataset: 
🔡 Dataset: 
resource
https://huggingface.co/datasets/google/reveal
dataset
google/reveal
https://huggingface.co/datasets/google/reveal
🔍 Today's pick in Interpretability & Analysis of LMs: A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains by @alonjacovi @yonatanbitton B. Bohnet J. Herzig @orhonovic M. Tseng M. Collins @roeeaharoni @mega

This work introduces a new methodology for human verification of reasoning chains and adopts it to annotate a dataset of chain-of-thought reasoning chains produced by 3 LMs. The annotated dataset, REVEAL, can be used to benchmark automatic verifiers of reasoning in LMs.

In their analysis, the authors find that LM-produced CoTs generally contain faulty steps, often leading to incorrect automatic verification. In particular, CoT-generating LMs are found to produce non-attributable reasoning steps often, and reasoning verifiers generally struggle to verify logical correctness.

📄 Paper: https://huggingface.co/papers/2402.00559
🔡 Dataset: https://huggingface.co/datasets/google/reveal
🤗
s3nh roeeaharoni alonjacovi osanseviero clem manu
👍
Yonatan-Bitton