Eliminate Hallucinations In RAG Applications With Context Filters

# Eliminating Hallucinations in RAG Applications with Context Filters

*A detailed walkthrough on improving RAG systems by filtering irrelevant context before it reaches your LLM*

 Introduction

Retrieval-Augmented Generation (RAG) applications have become a popular way to ground large language models (LLMs) in specific knowledge bases. However, these systems are prone to hallucinations when they receive irrelevant context. In this post, we'll explore how using context filters can dramatically improve RAG performance by ensuring only the most relevant information reaches your LLM.

This article is based on a presentation by Josh, a developer advocate for Snowflake and maintainer of TruLens, an open-source project for evaluating LLM applications.

 Understanding RAG Failure Modes

RAG applications can fail at multiple points in their workflow:

- **Query understanding**: Difficulties in parsing the user's true intent

- **LLM issues**: Feeding too much context or irrelevant context leading to hallucinations

- **Data challenges**: Selecting appropriate chunking methods and parsing approaches

- **Retrieval problems**: Choosing the right retriever and embedding methods for your specific data

The retrieval component is particularly critical because RAG follows a "garbage in, garbage out" principle. If we feed irrelevant context to our LLM, we'll get poor responses regardless of how powerful the model is.

The Context Filter Solution

Context filters act as guardrails sitting between your retrieval system and your LLM. They:

1. Evaluate each context chunk for relevance to the query

2. Score each chunk on a relevance scale (e.g., 0 to 1)

3. Filter out chunks scoring below a specified threshold

4. Feed only the most relevant chunks to the LLM

Here's a practical example:

Imagine a user asks: "How often are environmental KPIs assessed?" Our retrieval system returns three context chunks about environmental practices. Without filtering, the LLM might respond with a vague, uncertain answer combining information from all chunks.

However, if we analyze these chunks with a context filter and find that only one has a high relevance score (0.8), we can drop the less relevant chunks (scoring 0.2) and just pass the relevant one to the LLM. The result is a much clearer, more accurate response: "Environmental KPIs are assessed annually through audits."

Implementing Context Filters with Feedback Functions

To implement context filters, we need a way to score the relevance of each context chunk. This is done through "feedback functions" that take the query and context as input and return a relevance score.

These feedback functions can be:

- Human evaluators giving ratings

- AI judges (LLMs) scoring relevance

To implement a context filter workflow:

1. **Choose your model**: Select the LLM that will act as your judge (OpenAI, Anthropic, open-source options, etc.)

2. **Define evaluation criteria**: Specify how to determine relevance scores

3. **Apply the filter**: Add a context filter decorator to your retrieval function to automatically filter results

Practical Implementation

Let's look at how to implement context filters using TruLens:

```python

# Basic imports

from trulens import context_filter

from trulens.feedback import Feedback

from trulens.providers.openai import OpenAI

# Set up the feedback function with the model

provider = OpenAI(model_engine="gpt-4o")

feedback_function = provider.context_relevance

# Apply the context filter to your retrieval method

@context_filter(feedback_function, threshold=0.5)

def retrieve(query):

    # Your existing retrieval logic

    return retrieved_documents

```

When applied, this decorator will:

- Run the feedback function in parallel against all retrieved documents

- Score each document's relevance to the query

- Filter out documents scoring below your threshold

- Return only the most relevant documents

Performance Considerations

Using an LLM as a judge adds some latency to your application. In testing different models:

- GPT-4o took around 2.3 seconds to apply the filter

- Claude Mini was faster at about 1.7 seconds

- Groq running Llama 3 8B matched the speed of no filter at 1.7 seconds

- Local models running on consumer hardware were slower but required no API calls

The choice depends on your latency requirements, cost considerations, and accuracy needs. Smaller, faster models can often perform well for relevance judgments even if they're not suitable for your primary generation tasks.

 Visualizing the Benefits

Using TruLens's tracing capabilities, we can see the impact of context filters. In a sample application without filters, all four retrieved context chunks were passed to the LLM, with only one being truly relevant. With the context filter enabled, only the single relevant chunk was passed forward, resulting in:

- Cleaner, more accurate responses

- Reduced token usage

- Lower chance of hallucination

- Faster end-to-end completion times

Common Questions About Context Filters

**What is hallucination in this context?**  

Hallucination occurs when an LLM states facts not directly provided in the given context. The goal is to ensure the model only uses information explicitly provided.

**What causes hallucinations?**  

Common causes include:

- Mixing relevant and irrelevant context

- Underpowered LLMs or ones with high temperature settings

- Queries for which no relevant context exists in the knowledge base

**What if no context passes the threshold?**  

You can implement a fallback response that informs the user no relevant information was found, rather than sending an empty context to the LLM.

**Are there alternatives to using LLMs for scoring relevance?**  

Yes, cosine similarity is a faster alternative, though potentially less accurate. Other libraries like "Guardrails" and "Ragas" offer similar capabilities.

**How consistent are the scores across different models?**  

Scoring varies between models. GPT-4 tends to be harsher in its relevance judgments than smaller models. Calibration against human judgments can help determine appropriate thresholds for each model.

Conclusion

Context filters offer a powerful way to improve RAG applications by ensuring only truly relevant information reaches your LLM. This approach reduces hallucinations, improves response quality, and can even reduce costs by limiting the context size.

By implementing context filters as part of your RAG pipeline, you create a more robust system that delivers more accurate, reliable answers to your users.

The TruLens library makes it easy to add these capabilities to existing systems with minimal code changes. Whether you're using commercial APIs or open-source models, context filters can significantly enhance your RAG applications.

---