Reranking is a crucial step in Retrieval-Augmented Generation (RAG) systems that aims to improve the relevance and quality of retrieved documents. It involves reassessing and reordering initially retrieved documents to ensure that the most pertinent information is prioritized for subsequent processing or presentation.
Motivation
The primary motivation for reranking in RAG systems is to overcome limitations of initial retrieval methods, which often rely on simpler similarity metrics. Reranking allows for more sophisticated relevance assessment, taking into account nuanced relationships between queries and documents that might be missed by traditional retrieval techniques. This process aims to enhance the overall performance of RAG systems by ensuring that the most relevant information is used in the generation phase.
Key Components
Reranking systems typically include the following components:
- Initial Retriever: Often a vector store using embedding-based similarity search.
- Reranking Model: This can be either:
- A Large Language Model (LLM) for scoring relevance
- A Cross-Encoder model specifically trained for relevance assessment
- Scoring Mechanism: A method to assign relevance scores to documents
- Sorting and Selection Logic: To reorder documents based on new scores
Method Details
The reranking process generally follows these steps:
- Initial Retrieval: Fetch an initial set of potentially relevant documents.
- Pair Creation: Form query-document pairs for each retrieved document.
- Scoring:
- LLM Method: Use prompts to ask the LLM to rate document relevance.
- Cross-Encoder Method: Feed query-document pairs directly into the model.
- Score Interpretation: Parse and normalize the relevance scores.
- Reordering: Sort documents based on their new relevance scores.
- Selection: Choose the top K documents from the reordered list.
Benefits of this Approach
Reranking offers several advantages:
- Improved Relevance: By using more sophisticated models, reranking can capture subtle relevance factors.
- Flexibility: Different reranking methods can be applied based on specific needs and resources.
- Enhanced Context Quality: Providing more relevant documents to the RAG system improves the quality of generated responses.
- Reduced Noise: Reranking helps filter out less relevant information, focusing on the most pertinent content.
Reranking is a powerful technique in RAG systems that significantly enhances the quality of retrieved information. Whether using LLM-based scoring or specialized Cross-Encoder models, reranking allows for more nuanced and accurate assessment of document relevance. This improved relevance translates directly to better performance in downstream tasks, making reranking an essential component in advanced RAG implementations.
The choice between LLM-based and Cross-Encoder reranking methods depends on factors such as required accuracy, available computational resources, and specific application needs. Both approaches offer substantial improvements over basic retrieval methods and contribute to the overall effectiveness of RAG systems.
Import relevant libraries
import os
import sys
from dotenv import load_dotenv
from typing import List
from llama_index.core import Document
from llama_index.core import Settings
from llama_index.embeddings.openai import OpenAIEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core.readers import SimpleDirectoryReader
from llama_index.vector_stores.faiss import FaissVectorStore
from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core import VectorStoreIndex
from llama_index.core.postprocessor import SentenceTransformerRerank, LLMRerank
from llama_index.core import QueryBundle
import faiss
sys.path.append(os.path.abspath(os.path.join(os.getcwd(), '..'))) # Add the parent directory to the path sicnce we work with notebooks
# Load environment variables from a .env file
load_dotenv()
# Set the OpenAI API key environment variable
os.environ["OPENAI_API_KEY"] = os.getenv('OPENAI_API_KEY')
# Llamaindex global settings for llm and embeddings
EMBED_DIMENSION=512
Settings.llm = OpenAI(model="gpt-3.5-turbo")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small", dimensions=EMBED_DIMENSION)
Read docs
path = "../data/"
reader = SimpleDirectoryReader(input_dir=path, required_exts=['.pdf'])
documents = reader.load_data()
Create a vector store
# Create FaisVectorStore to store embeddings
fais_index = faiss.IndexFlatL2(EMBED_DIMENSION)
vector_store = FaissVectorStore(faiss_index=fais_index)
Ingestion Pipeline
base_pipeline = IngestionPipeline(
transformations=[SentenceSplitter()],
vector_store=vector_store,
documents=documents
)
nodes = base_pipeline.run()
Querying
Method 1: LLM based reranking the retrieved documents
# Create vector index from base nodes
index = VectorStoreIndex(nodes)
query_engine_w_llm_rerank = index.as_query_engine(
similarity_top_k=10,
node_postprocessors=[
LLMRerank(
top_n=5
)
],
)
resp = query_engine_w_llm_rerank.query("What are the impacts of climate change on biodiversity?")
print(resp)
Example that demonstrates why we should use reranking
chunks = [
"The capital of France is great.",
"The capital of France is huge.",
"The capital of France is beautiful.",
"""Have you ever visited Paris? It is a beautiful city where you can eat delicious food and see the Eiffel Tower. I really enjoyed all the cities in france, but its capital with the Eiffel Tower is my favorite city.""",
"I really enjoyed my trip to Paris, France. The city is beautiful and the food is delicious. I would love to visit again. Such a great capital city."
]
docs = [Document(page_content=sentence) for sentence in chunks]
def compare_rag_techniques(query: str, docs: List[Document] = docs) -> None:
docs = [Document(text=sentence) for sentence in chunks]
index = VectorStoreIndex.from_documents(docs)
print("Comparison of Retrieval Techniques")
print("==================================")
print(f"Query: {query}\n")
print("Baseline Retrieval Result:")
baseline_docs = index.as_retriever(similarity_top_k=5).retrieve(query)
for i, doc in enumerate(baseline_docs[:2]): # Get only the first two retrieved docs
print(f"\nDocument {i+1}:")
print(doc.text)
print("\nAdvanced Retrieval Result:")
reranker = LLMRerank(
top_n=2,
)
advanced_docs = reranker.postprocess_nodes(
baseline_docs,
QueryBundle(query)
)
for i, doc in enumerate(advanced_docs):
print(f"\nDocument {i+1}:")
print(doc.text)
query = "what is the capital of france?"
compare_rag_techniques(query, docs)
Comparison of Retrieval Techniques ================================== Query: what is the capital of france? Baseline Retrieval Result: Document 1: The capital of France is great. Document 2: The capital of France is huge. Advanced Retrieval Result: Document 1: Have you ever visited Paris? It is a beautiful city where you can eat delicious food and see the Eiffel Tower. I really enjoyed all the cities in france, but its capital with the Eiffel Tower is my favorite city. Document 2: I really enjoyed my trip to Paris, France. The city is beautiful and the food is delicious. I would love to visit again. Such a great capital city.
Method 2: Cross Encoder models
LlamaIndex has builtin support for SBERT models that can be used directly as node postprocessor.
query_engine_w_cross_encoder = index.as_query_engine(
similarity_top_k=10,
node_postprocessors=[
SentenceTransformerRerank(
model='cross-encoder/ms-marco-MiniLM-L-6-v2',
top_n=5
)
],
)
resp = query_engine_w_cross_encoder.query("What are the impacts of climate change on biodiversity?")
print(resp)
