The Impact of Symbolic Representations on In-context Learning for Few-shot Reasoning

. Abstract Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations (or “chain-of-thought” (CoT)) for in-context learning. On the other hand, these reasoning tasks are usually pre-sumed to be more approachable for symbolic programming. To make progress towards understanding in-context learning, we curate synthetic datasets containing equivalent (natural, symbolic) data pairs, where symbolic examples contain ﬁrst-order logic rules and predicates from knowledge bases (KBs). Then we revisit neuro-symbolic approaches and use Language Models as Logic Program-mer (LMLP) that learns from demonstrations containing logic rules and corresponding examples to iteratively reason over KBs, recov-ering Prolog’s backward chaining algorithm. Comprehensive experiments are included to systematically compare LMLP with CoT in deductive reasoning settings, showing that LMLP enjoys more than 25% higher accuracy than COT on length generalization bench-marks even with fewer parameters. 1