Faithful Chain-Of-Though Bridging The Faithfulness Gap In LLMs

Faithful Chain of Thought Reasoning: Bridging the Faithfulness Gap in Language Models

Introduction and Motivation

Chain of Thought (CoT) prompting has emerged as a highly effective in-context learning technique for reasoning tasks. Unlike standard prompting where language models only generate a final answer, CoT prompting encourages models to generate a reasoning chain before producing the final answer. This approach has significantly improved language model performance on complex reasoning tasks like mathematics and multi-hop question answering.

However, an important question remains: Do CoT reasoning chains actually provide good explanations of how the model reaches its answers? The evidence suggests this isn't always the case. In many instances, the final answer doesn't follow from the reasoning chain, meaning that CoT prompting doesn't necessarily provide a faithful explanation of the model's reasoning process. A faithful explanation should accurately reflect how the model arrives at its final prediction, but language models don't always "say what they think" when predicting answers.

The Risks of Unfaithful Explanations

Unfaithful explanations pose several risks:

- They can give a false impression that models are self-interpretable

- They may induce users to overtrust model outputs

- This is particularly dangerous in high-stakes domains such as law and healthcare

Faithful Chain of Thought: A New Approach

To address this faithfulness gap, researchers from the University of Pennsylvania (Veronica Qing Xu, Sherry Yang, Adam Stein, Harry Desmond, Eric Mitchell, Mariana Bernagozzi, and Chris Callison-Burch) developed a framework called Faithful Chain of Thought reasoning, which decomposes reasoning tasks into two distinct stages:

1. **Translation Stage**: A language model (the "translator") generates a reasoning chain that interleaves natural language and symbolic language. The natural language portion decomposes the original question into simpler sub-problems, while the symbolic language addresses each sub-problem.

2. **Problem-Solving Stage**: A deterministic solver executes the reasoning chain to derive the answer.

This process is faithful by construction because the answer is the result of deterministically executing the reasoning chain, ensuring the chain accurately reflects how the answer is derived.

Application Across Various Reasoning Tasks

The Faithful Chain of Thought approach can be applied to multiple reasoning problems by integrating different choices of symbolic language and solvers:

1. Math Reasoning

For math word problems, the translator generates a reasoning chain with natural language sub-problems and Python code to solve each one. A Python interpreter then executes this chain to derive the answer.

**Example**: "If there are three trees in a parking lot and two more arrive, how many trees are there in total?"

- The language model decomposes this into sub-problems and generates Python code for each

- The Python interpreter executes the code to derive the answer: 5

2. Multi-hop Question Answering

For questions requiring multiple steps of reasoning, the symbolic language chosen is Datalog, a declarative logic programming language.

Example: "Would an apple sink in water?"

- The question is decomposed into simpler sub-questions (e.g., "What's the density of an apple?" and "What's the density of water?")

- Each sub-question is answered in Datalog

- A Datalog interpreter derives the final answer: No

3. Relational Inference

For stories involving family relationships, custom-defined logical expressions serve as the symbolic language.

Example: "Gabrielle drove her daughter Dorothy to the hospital, and Dorothy's son Vincent showed up shortly after. How is Vincent related to Gabrielle?"

- The task is decomposed into sub-questions with custom logical expressions

- A rule-based inference engine applies relationship rules (e.g., "one's daughter's son is one's grandson")

- Final answer: Vincent is Gabrielle's grandson

 4. Planning

For planning tasks, the symbolic language is PDDL (Planning Domain Definition Language).

Example: "I spilled my Coke on the table. Could you throw it away and bring something to clean it?"

- The translator decomposes the query into sub-goals and expresses them in PDDL

- A PDDL planner generates a sequence of executable actions to achieve these goals

 Experimental Results

The researchers compared their approach to several baselines:

- Standard prompting (generates only the answer)

- Chain of Thought prompting (generates a natural language reasoning chain)

- Least-to-Most prompting (decomposes questions into sub-questions but doesn't involve symbolic language)

Experiments used various OpenAI language models (GPT-3, Codex, ChatGPT, and GPT-4) and evaluated on 10 datasets across four domains. The number of shots was controlled (6-10) following previous work, and two decoding strategies were tested: greedy and self-consistency decoding.

Key Findings

1. **Superior Performance**: Faithful Chain of Thought outperformed all baselines on 8 out of 10 datasets using the same underlying language model (Codex). Average accuracy gains were:

   - 5% on math word problems

   - 2% on planning tasks

   - 4% on multi-hop question answering

   - 18% on relational inference

2. **State-of-the-Art Results**: Using ChatGPT and GPT-4, the approach achieved new state-of-the-art performance on 7 out of 10 datasets, demonstrating that improved interpretability doesn't require sacrificing performance.

3. **Robustness**: The approach proved robust to both the choice of exemplars and the phrasing of the prompt.

4. **Importance of the Solver**: When the solver component was removed (ablation study), performance dropped significantly across almost all datasets, highlighting that the solver is essential not only for interpretability but also for performance.

 Conclusion

Faithful Chain of Thought provides a two-stage reasoning framework that guarantees the reasoning chain faithfully explains how the model arrives at its answer. The approach outperforms existing baselines while providing faithful explanations, demonstrating a strong synergy between interpretability and performance.

The comprehensive analysis also revealed the robustness of the method, the critical role of the solver component, and identified some frequent error patterns where the approach still struggles.

This research represents an important step toward making language model reasoning both more effective and more transparent.