Propositions Chunking

On this page13 sections

Definition

This code implements the proposition chunking method, based on research from Tony Chen, et. al.. The system break downs the input text into propositions that are atomic, factual, self-contained, and concise in nature, encodes the propositions into a vectorstore, which can be later used for retrieval

Key Components

Document Chunking: Splitting a document into manageable pieces for analysis.
Proposition Generation: Using LLMs to break down document chunks into factual, self-contained propositions.
Proposition Quality Check: Evaluating generated propositions based on accuracy, clarity, completeness, and conciseness.
Embedding and Vector Store: Embedding both propositions and larger chunks of the document into a vector store for efficient retrieval.
Retrieval and Comparison: Testing the retrieval system with different query sizes and comparing results from the proposition-based model with the larger chunk-based model.

Propositions Chunking

Motivation

The motivation behind the propositions chunking method is to build a system that breaks down a text document into concise, factual propositions for more granular and precise information retrieval. Using propositions allows for finer control and better handling of specific queries, particularly for extracting knowledge from detailed or complex texts. The comparison between using smaller proposition chunks and larger document chunks aims to evaluate the effectiveness of granular information retrieval.

Method Details

Loading Environment Variables: The code begins by loading environment variables (e.g., API keys for the LLM service) to ensure that the system can access the necessary resources.
Document Chunking:
- The input document is split into smaller pieces (chunks) using RecursiveCharacterTextSplitter. This ensures that each chunk is of manageable size for the LLM to process.
Proposition Generation:
- Propositions are generated from each chunk using an LLM (in this case, “llama-3.1-70b-versatile”). The output is structured as a list of factual, self-contained statements that can be understood without additional context.
Quality Check:
- A second LLM evaluates the quality of the propositions by scoring them on accuracy, clarity, completeness, and conciseness. Propositions that meet the required thresholds in all categories are retained.
Embedding Propositions:
- Propositions that pass the quality check are embedded into a vector store using the OllamaEmbeddings model. This allows for similarity-based retrieval of propositions when queries are made.
Retrieval and Comparison:
- Two retrieval systems are built: one using the proposition-based chunks and another using larger document chunks. Both are tested with several queries to compare their performance and the precision of the returned results.

Benefits

Granularity: By breaking the document into small factual propositions, the system allows for highly specific retrieval, making it easier to extract precise answers from large or complex documents.
Quality Assurance: The use of a quality-checking LLM ensures that the generated propositions meet specific standards, improving the reliability of the retrieved information.
Flexibility in Retrieval: The comparison between proposition-based and larger chunk-based retrieval allows for evaluating the trade-offs between granularity and broader context in search results.

Implementation

Proposition Generation: The LLM is used in conjunction with a custom prompt to generate factual statements from the document chunks.
Quality Checking: The generated propositions are passed through a grading system that evaluates accuracy, clarity, completeness, and conciseness.
Vector Store Integration: Propositions are stored in a FAISS vector store after being embedded using a pre-trained embedding model, allowing for efficient similarity-based search and retrieval.
Query Testing: Multiple test queries are made to the vector stores (proposition-based and larger chunks) to compare the retrieval performance.

This code presents a robust method for breaking down a document into self-contained propositions using LLMs. The system performs a quality check on each proposition, embeds them in a vector store, and retrieves the most relevant information based on user queries. The ability to compare granular propositions against larger document chunks provides insight into which method yields more accurate or useful results for different types of queries. The approach emphasizes the importance of high-quality proposition generation and retrieval for precise information extraction from complex documents.

In [1]:

### LLMs
import os
from dotenv import load_dotenv

# Load environment variables from '.env' file
load_dotenv()

os.environ['GROQ_API_KEY'] = os.getenv('GROQ_API_KEY') # For LLM

Test Document

In [2]:

sample_content = """Paul Graham's essay "Founder Mode," published in September 2024, challenges conventional wisdom about scaling startups, arguing that founders should maintain their unique management style rather than adopting traditional corporate practices as their companies grow.
Conventional Wisdom vs. Founder Mode
The essay argues that the traditional advice given to growing companies—hiring good people and giving them autonomy—often fails when applied to startups.
This approach, suitable for established companies, can be detrimental to startups where the founder's vision and direct involvement are crucial. "Founder Mode" is presented as an emerging paradigm that is not yet fully understood or documented, contrasting with the conventional "manager mode" often advised by business schools and professional managers.
Unique Founder Abilities
Founders possess unique insights and abilities that professional managers do not, primarily because they have a deep understanding of their company's vision and culture.
Graham suggests that founders should leverage these strengths rather than conform to traditional managerial practices. "Founder Mode" is an emerging paradigm that is not yet fully understood or documented, with Graham hoping that over time, it will become as well-understood as the traditional manager mode, allowing founders to maintain their unique approach even as their companies scale.
Challenges of Scaling Startups
As startups grow, there is a common belief that they must transition to a more structured managerial approach. However, many founders have found this transition problematic, as it often leads to a loss of the innovative and agile spirit that drove the startup's initial success.
Brian Chesky, co-founder of Airbnb, shared his experience of being advised to run the company in a traditional managerial style, which led to poor outcomes. He eventually found success by adopting a different approach, influenced by how Steve Jobs managed Apple.
Steve Jobs' Management Style
Steve Jobs' management approach at Apple served as inspiration for Brian Chesky's "Founder Mode" at Airbnb. One notable practice was Jobs' annual retreat for the 100 most important people at Apple, regardless of their position on the organizational chart
. This unconventional method allowed Jobs to maintain a startup-like environment even as Apple grew, fostering innovation and direct communication across hierarchical levels. Such practices emphasize the importance of founders staying deeply involved in their companies' operations, challenging the traditional notion of delegating responsibilities to professional managers as companies scale.
"""

Chunking

In [3]:

### Build Index
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import OllamaEmbeddings
# Set embeddings
embedding_model = OllamaEmbeddings(model='nomic-embed-text:v1.5', show_progress=True)
# docs
docs_list = [Document(page_content=sample_content, metadata={"Title": "Paul Graham's Founder Mode Essay", "Source": "https://www.perplexity.ai/page/paul-graham-s-founder-mode-ess-t9TCyvkqRiyMQJWsHr0fnQ"})]
# Split
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=200, chunk_overlap=50
)
doc_splits = text_splitter.split_documents(docs_list)

In [4]:

for i, doc in enumerate(doc_splits):
    doc.metadata['chunk_id'] = i+1 ### adding chunk id

Generate Propositions

In [5]:

from typing import List
from langchain_core.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_groq import ChatGroq
# Data model
class GeneratePropositions(BaseModel):
    """List of all the propositions in a given document"""

    propositions: List[str] = Field(
        description="List of propositions (factual, self-contained, and concise information)"
    )
# LLM with function call
llm = ChatGroq(model="llama-3.1-70b-versatile", temperature=0)
structured_llm= llm.with_structured_output(GeneratePropositions)

# Few shot prompting --- We can add more examples to make it good
proposition_examples = [
    {"document": 
        "In 1969, Neil Armstrong became the first person to walk on the Moon during the Apollo 11 mission.", 
     "propositions": 
        "['Neil Armstrong was an astronaut.', 'Neil Armstrong walked on the Moon in 1969.', 'Neil Armstrong was the first person to walk on the Moon.', 'Neil Armstrong walked on the Moon during the Apollo 11 mission.', 'The Apollo 11 mission occurred in 1969.']"
    },
]
example_proposition_prompt = ChatPromptTemplate.from_messages(
    [
        ("human", "{document}"),
        ("ai", "{propositions}"),
    ]
)
few_shot_prompt = FewShotChatMessagePromptTemplate(
    example_prompt = example_proposition_prompt,
    examples = proposition_examples,
)
# Prompt
system = """Please break down the following text into simple, self-contained propositions. Ensure that each proposition meets the following criteria:

    1. Express a Single Fact: Each proposition should state one specific fact or claim.
    2. Be Understandable Without Context: The proposition should be self-contained, meaning it can be understood without needing additional context.
    3. Use Full Names, Not Pronouns: Avoid pronouns or ambiguous references; use full entity names.
    4. Include Relevant Dates/Qualifiers: If applicable, include necessary dates, times, and qualifiers to make the fact precise.
    5. Contain One Subject-Predicate Relationship: Focus on a single subject and its corresponding action or attribute, without conjunctions or multiple clauses."""
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", system),
        few_shot_prompt,
        ("human", "{document}"),
    ]
)
proposition_generator = prompt | structured_llm

In [6]:

propositions = [] # Store all the propositions from the document

for i in range(len(doc_splits)):
    response = proposition_generator.invoke({"document": doc_splits[i].page_content}) # Creating proposition
    for proposition in response.propositions:
        propositions.append(Document(page_content=proposition, metadata={"Title": "Paul Graham's Founder Mode Essay", "Source": "https://www.perplexity.ai/page/paul-graham-s-founder-mode-ess-t9TCyvkqRiyMQJWsHr0fnQ", "chunk_id": i+1}))

Quality Check

In [7]:

# Data model
class GradePropositions(BaseModel):
    """Grade a given proposition on accuracy, clarity, completeness, and conciseness"""

    accuracy: int = Field(
        description="Rate from 1-10 based on how well the proposition reflects the original text."
    )   
    clarity: int = Field(
        description="Rate from 1-10 based on how easy it is to understand the proposition without additional context."
    )
    completeness: int = Field(
        description="Rate from 1-10 based on whether the proposition includes necessary details (e.g., dates, qualifiers)."
    )
    conciseness: int = Field(
        description="Rate from 1-10 based on whether the proposition is concise without losing important information."
    )
# LLM with function call
llm = ChatGroq(model="llama-3.1-70b-versatile", temperature=0)
structured_llm= llm.with_structured_output(GradePropositions)
# Prompt
evaluation_prompt_template = """
Please evaluate the following proposition based on the criteria below:
- **Accuracy**: Rate from 1-10 based on how well the proposition reflects the original text.
- **Clarity**: Rate from 1-10 based on how easy it is to understand the proposition without additional context.
- **Completeness**: Rate from 1-10 based on whether the proposition includes necessary details (e.g., dates, qualifiers).
- **Conciseness**: Rate from 1-10 based on whether the proposition is concise without losing important information.
Example
Docs: In 1969, Neil Armstrong became the first person to walk on the Moon during the Apollo 11 mission.
Propositons_1: Neil Armstrong was an astronaut.
Evaluation_1: "accuracy": 10, "clarity": 10, "completeness": 10, "conciseness": 10
Propositons_2: Neil Armstrong walked on the Moon in 1969.
Evaluation_3: "accuracy": 10, "clarity": 10, "completeness": 10, "conciseness": 10
Propositons_3: Neil Armstrong was the first person to walk on the Moon.
Evaluation_3: "accuracy": 10, "clarity": 10, "completeness": 10, "conciseness": 10
Propositons_4: Neil Armstrong walked on the Moon during the Apollo 11 mission.
Evaluation_4: "accuracy": 10, "clarity": 10, "completeness": 10, "conciseness": 10
Propositons_5: The Apollo 11 mission occurred in 1969.
Evaluation_5: "accuracy": 10, "clarity": 10, "completeness": 10, "conciseness": 10
Format:
Proposition: "{proposition}"
Original Text: "{original_text}"
"""
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", evaluation_prompt_template),
        ("human", "{proposition}, {original_text}"),
    ]
)
proposition_evaluator = prompt | structured_llm

In [8]:

# Define evaluation categories and thresholds
evaluation_categories = ["accuracy", "clarity", "completeness", "conciseness"]
thresholds = {"accuracy": 7, "clarity": 7, "completeness": 7, "conciseness": 7}
# Function to evaluate proposition
def evaluate_proposition(proposition, original_text):
    response = proposition_evaluator.invoke({"proposition": proposition, "original_text": original_text})   
    # Parse the response to extract scores
    scores = {"accuracy": response.accuracy, "clarity": response.clarity, "completeness": response.completeness, "conciseness": response.conciseness}  # Implement function to extract scores from the LLM response
    return scores
# Check if the proposition passes the quality check
def passes_quality_check(scores):
    for category, score in scores.items():
        if score < thresholds[category]:
            return False
    return True
evaluated_propositions = [] # Store all the propositions from the document
# Loop through generated propositions and evaluate them
for idx, proposition in enumerate(propositions):
    scores = evaluate_proposition(proposition.page_content, doc_splits[proposition.metadata['chunk_id'] - 1].page_content)
    if passes_quality_check(scores):
        # Proposition passes quality check, keep it
        evaluated_propositions.append(proposition)
    else:
        # Proposition fails, discard or flag for further review
        print(f"{idx+1}) Propostion: {proposition.page_content} \n Scores: {scores}")
        print("Fail")

17) Propostion: Startups often transition to a more structured managerial approach as they grow. 
 Scores: {'accuracy': 8, 'clarity': 9, 'completeness': 6, 'conciseness': 8}
Fail
31) Propostion: Delegating responsibilities to professional managers is not always the best approach as companies scale. 
 Scores: {'accuracy': 10, 'clarity': 10, 'completeness': 8, 'conciseness': 6}
Fail

Embedding propositions in a vectorstore

In [9]:

# Add to vectorstore
vectorstore_propositions = FAISS.from_documents(evaluated_propositions, embedding_model)
retriever_propositions = vectorstore_propositions.as_retriever(
                search_type="similarity",
                search_kwargs={'k': 4}, # number of documents to retrieve
            )

OllamaEmbeddings: 100%|██████████| 29/29 [00:08<00:00,  3.62it/s]

In [10]:

query = "Who's management approach served as inspiartion for Brian Chesky's \"Founder Mode\" at Airbnb?"
res_proposition = retriever_propositions.invoke(query)

OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00,  5.39it/s]

In [11]:

for i, r in enumerate(res_proposition):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}")

1) Content: Brian Chesky was advised to run Airbnb in a traditional managerial style. --- Chunk_id: 3
2) Content: Brian Chesky adopted a different approach to running Airbnb. --- Chunk_id: 3
3) Content: Brian Chesky is a co-founder of Airbnb. --- Chunk_id: 3
4) Content: Steve Jobs' management style at Apple influenced Brian Chesky's approach. --- Chunk_id: 3

Comparing performance with larger chunks size

In [12]:

# Add to vectorstore_larger_
vectorstore_larger = FAISS.from_documents(doc_splits, embedding_model)
retriever_larger = vectorstore_larger.as_retriever(
                search_type="similarity",
                search_kwargs={'k': 4}, # number of documents to retrieve
            )

OllamaEmbeddings: 100%|██████████| 3/3 [00:00<00:00,  5.35it/s]

In [13]:

res_larger = retriever_larger.invoke(query)

OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00,  6.64it/s]

In [14]:

for i, r in enumerate(res_larger):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}")

1) Content: Brian Chesky, co-founder of Airbnb, shared his experience of being advised to run the company in a traditional managerial style, which led to poor outcomes. He eventually found success by adopting a different approach, influenced by how Steve Jobs managed Apple.
Steve Jobs' Management Style
Steve Jobs' management approach at Apple served as inspiration for Brian Chesky's "Founder Mode" at Airbnb. One notable practice was Jobs' annual retreat for the 100 most important people at Apple, regardless of their position on the organizational chart
. This unconventional method allowed Jobs to maintain a startup-like environment even as Apple grew, fostering innovation and direct communication across hierarchical levels. Such practices emphasize the importance of founders staying deeply involved in their companies' operations, challenging the traditional notion of delegating responsibilities to professional managers as companies scale. --- Chunk_id: 3
2) Content: Unique Founder Abilities
Founders possess unique insights and abilities that professional managers do not, primarily because they have a deep understanding of their company's vision and culture.
Graham suggests that founders should leverage these strengths rather than conform to traditional managerial practices. "Founder Mode" is an emerging paradigm that is not yet fully understood or documented, with Graham hoping that over time, it will become as well-understood as the traditional manager mode, allowing founders to maintain their unique approach even as their companies scale.
Challenges of Scaling Startups
As startups grow, there is a common belief that they must transition to a more structured managerial approach. However, many founders have found this transition problematic, as it often leads to a loss of the innovative and agile spirit that drove the startup's initial success. --- Chunk_id: 2
3) Content: Paul Graham's essay "Founder Mode," published in September 2024, challenges conventional wisdom about scaling startups, arguing that founders should maintain their unique management style rather than adopting traditional corporate practices as their companies grow.
Conventional Wisdom vs. Founder Mode
The essay argues that the traditional advice given to growing companies—hiring good people and giving them autonomy—often fails when applied to startups.
This approach, suitable for established companies, can be detrimental to startups where the founder's vision and direct involvement are crucial. "Founder Mode" is presented as an emerging paradigm that is not yet fully understood or documented, contrasting with the conventional "manager mode" often advised by business schools and professional managers.
Unique Founder Abilities
Founders possess unique insights and abilities that professional managers do not, primarily because they have a deep understanding of their company's vision and culture. --- Chunk_id:

Testing

Test – 1

In [15]:

test_query_1 = "what is the essay \"Founder Mode\" about?"
res_proposition = retriever_propositions.invoke(test_query_1)
res_larger = retriever_larger.invoke(test_query_1)

OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00,  6.29it/s]
OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00,  8.06it/s]

In [16]:

for i, r in enumerate(res_proposition):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}"

1) Content: Founder Mode is an emerging paradigm that is not yet fully understood or documented. --- Chunk_id: 2
2) Content: Founder Mode is not yet fully understood or documented. --- Chunk_id: 1
3) Content: Founder Mode is an emerging paradigm. --- Chunk_id: 1
4) Content: Paul Graham's essay 'Founder Mode' challenges conventional wisdom about scaling startups. --- Chunk_id: 1

In [17]:

for i, r in enumerate(res_larger):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}")

1) Content: Paul Graham's essay "Founder Mode," published in September 2024, challenges conventional wisdom about scaling startups, arguing that founders should maintain their unique management style rather than adopting traditional corporate practices as their companies grow.
Conventional Wisdom vs. Founder Mode
The essay argues that the traditional advice given to growing companies—hiring good people and giving them autonomy—often fails when applied to startups.
This approach, suitable for established companies, can be detrimental to startups where the founder's vision and direct involvement are crucial. "Founder Mode" is presented as an emerging paradigm that is not yet fully understood or documented, contrasting with the conventional "manager mode" often advised by business schools and professional managers.
Unique Founder Abilities
Founders possess unique insights and abilities that professional managers do not, primarily because they have a deep understanding of their company's vision and culture. --- Chunk_id: 1
2) Content: Unique Founder Abilities
Founders possess unique insights and abilities that professional managers do not, primarily because they have a deep understanding of their company's vision and culture.
Graham suggests that founders should leverage these strengths rather than conform to traditional managerial practices. "Founder Mode" is an emerging paradigm that is not yet fully understood or documented, with Graham hoping that over time, it will become as well-understood as the traditional manager mode, allowing founders to maintain their unique approach even as their companies scale.
Challenges of Scaling Startups
As startups grow, there is a common belief that they must transition to a more structured managerial approach. However, many founders have found this transition problematic, as it often leads to a loss of the innovative and agile spirit that drove the startup's initial success. --- Chunk_id: 2
3) Content: Brian Chesky, co-founder of Airbnb, shared his experience of being advised to run the company in a traditional managerial style, which led to poor outcomes. He eventually found success by adopting a different approach, influenced by how Steve Jobs managed Apple.
Steve Jobs' Management Style
Steve Jobs' management approach at Apple served as inspiration for Brian Chesky's "Founder Mode" at Airbnb. One notable practice was Jobs' annual retreat for the 100 most important people at Apple, regardless of their position on the organizational chart
. This unconventional method allowed Jobs to maintain a startup-like environment even as Apple grew, fostering innovation and direct communication across hierarchical levels. Such practices emphasize the importance of founders staying deeply involved in their companies' operations, challenging the traditional notion of delegating responsibilities to professional managers as companies scale. --- Chunk_id: 3

Test – 2

In [18]:

test_query_2 = "who is the co-founder of Airbnb?"
res_proposition = retriever_propositions.invoke(test_query_2)
res_larger = retriever_larger.invoke(test_query_2)

OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00,  3.22it/s]
OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00, 15.18it/s]

In [19]:

for i, r in enumerate(res_proposition):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}")

1) Content: Brian Chesky is a co-founder of Airbnb. --- Chunk_id: 3
2) Content: Brian Chesky adopted a different approach to running Airbnb. --- Chunk_id: 3
3) Content: Brian Chesky was advised to run Airbnb in a traditional managerial style. --- Chunk_id: 3
4) Content: Running Airbnb in a traditional managerial style led to poor outcomes. --- Chunk_id: 3

In [20]:

for i, r in enumerate(res_larger):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}")

1) Content: Brian Chesky, co-founder of Airbnb, shared his experience of being advised to run the company in a traditional managerial style, which led to poor outcomes. He eventually found success by adopting a different approach, influenced by how Steve Jobs managed Apple.
Steve Jobs' Management Style
Steve Jobs' management approach at Apple served as inspiration for Brian Chesky's "Founder Mode" at Airbnb. One notable practice was Jobs' annual retreat for the 100 most important people at Apple, regardless of their position on the organizational chart
. This unconventional method allowed Jobs to maintain a startup-like environment even as Apple grew, fostering innovation and direct communication across hierarchical levels. Such practices emphasize the importance of founders staying deeply involved in their companies' operations, challenging the traditional notion of delegating responsibilities to professional managers as companies scale. --- Chunk_id: 3
2) Content: Paul Graham's essay "Founder Mode," published in September 2024, challenges conventional wisdom about scaling startups, arguing that founders should maintain their unique management style rather than adopting traditional corporate practices as their companies grow.
Conventional Wisdom vs. Founder Mode
The essay argues that the traditional advice given to growing companies—hiring good people and giving them autonomy—often fails when applied to startups.
This approach, suitable for established companies, can be detrimental to startups where the founder's vision and direct involvement are crucial. "Founder Mode" is presented as an emerging paradigm that is not yet fully understood or documented, contrasting with the conventional "manager mode" often advised by business schools and professional managers.
Unique Founder Abilities
Founders possess unique insights and abilities that professional managers do not, primarily because they have a deep understanding of their company's vision and culture. --- Chunk_id: 1
3) Content: Unique Founder Abilities
Founders possess unique insights and abilities that professional managers do not, primarily because they have a deep understanding of their company's vision and culture.
Graham suggests that founders should leverage these strengths rather than conform to traditional managerial practices. "Founder Mode" is an emerging paradigm that is not yet fully understood or documented, with Graham hoping that over time, it will become as well-understood as the traditional manager mode, allowing founders to maintain their unique approach even as their companies scale.
Challenges of Scaling Startups
As startups grow, there is a common belief that they must transition to a more structured managerial approach. However, many founders have found this transition problematic, as it often leads to a loss of the innovative and agile spirit that drove the startup's initial success. --- Chunk_id: 2

Test – 3

In [21]:

test_query_3 = "when was the essay \"founder mode\" published?"
res_proposition = retriever_propositions.invoke(test_query_3)
res_larger = retriever_larger.invoke(test_query_3)

OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00, 10.09it/s]
OllamaEmbeddings: 100%|██████████| 1/1 [00:00<00:00,  7.71it/s]

In [22]:

for i, r in enumerate(res_proposition):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}")

1) Content: Paul Graham published an essay called 'Founder Mode' in September 2024. --- Chunk_id: 1
2) Content: Founder Mode is an emerging paradigm. --- Chunk_id: 1
3) Content: Founder Mode is an emerging paradigm that is not yet fully understood or documented. --- Chunk_id: 2
4) Content: Founder Mode is not yet fully understood or documented. --- Chunk_id: 1

In [23]:

for i, r in enumerate(res_larger):
    print(f"{i+1}) Content: {r.page_content} --- Chunk_id: {r.metadata['chunk_id']}")

Comparison

Aspect	Proposition-Based Retrieval	Simple Chunk Retrieval
Precision in Response	High: Delivers focused and direct answers.	Medium: Provides more context but may include irrelevant information.
Clarity and Brevity	High: Clear and concise, avoids unnecessary details.	Medium: More comprehensive but can be overwhelming.
Contextual Richness	Low: May lack context, focusing on specific propositions.	High: Provides additional context and details.
Comprehensiveness	Low: May omit broader context or supplementary details.	High: Offers a more complete view with extensive information.
Narrative Flow	Medium: Can be fragmented or disjointed.	High: Preserves the logical flow and coherence of the original document.
Information Overload	Low: Less likely to overwhelm with excess information.	High: Risk of overwhelming the user with too much information.
Use Case Suitability	Best for quick, factual queries.	Best for complex queries requiring in-depth understanding.
Efficiency	High: Provides quick, targeted responses.	Medium: May require more effort to sift through additional content.
Specificity	High: Precise and targeted responses.	Medium: Answers may be less targeted due to inclusion of broader context.

Key Components

Motivation

Method Details

Benefits

Implementation

Test Document

Chunking

Generate Propositions

Quality Check

Embedding propositions in a vectorstore

Comparing performance with larger chunks size

Testing

Test – 1

Test – 2

Test – 3

Comparison

Trust and publishing policies

About RX Harun

People behind the mission

Patient pathways

Popular searches

Recent searches

Key Components

Motivation

Method Details

Benefits

Implementation

Test Document

Chunking

Generate Propositions

Quality Check

Embedding propositions in a vectorstore

Comparing performance with larger chunks size

Testing

Test – 1

Test – 2

Test – 3

Comparison

To Get Daily Health Newsletter