RAPTOR: Enhancing Retrieval-Augmented Language Models with Tree-Organized Knowledge

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks. Their immense size allows them to encode vast amounts of world knowledge within their parameters, serving as powerful standalone knowledge stores. However, this parametric knowledge has inherent limitations. LLMs can struggle with highly domain-specific information, their knowledge is static and quickly becomes outdated in a changing world, and the source of their internal knowledge is often opaque, making fact-checking and provenance tracking challenging.

The Rise of Retrieval Augmentation

To address these limitations, retrieval-augmented language models (RALMs) have emerged as a prominent solution. This approach combines the generative power of LLMs with external, up-to-date knowledge bases. Instead of relying solely on internal parameters, RALMs query an external retrieval system to fetch relevant documents or text snippets pertinent to a given query or context. This retrieved information is then provided to the LLM as supplementary context, enabling it to generate more accurate, current, and grounded responses. This method offers significant advantages: it allows models to adapt to new information without costly retraining, provides access to long-tail knowledge, and offers greater transparency by allowing users to trace generated information back to its source document.

Traditional retrieval systems used in RALMs typically index large text corpora by splitting them into smaller, contiguous chunks, often paragraphs or fixed-size segments. During inference, the system retrieves a small number of these chunks that are deemed most relevant to the user's query based on similarity metrics, usually using dense vector embeddings. These retrieved chunks are then passed to the LLM as part of the input prompt.

The Challenge with Long Documents and Complex Queries

While effective for many tasks, the reliance on retrieving only a few short, contiguous text chunks presents a significant limitation, particularly when dealing with long documents or questions that require integrating information from multiple, potentially non-adjacent sections of a text. Complex questions often demand a holistic understanding of the entire document context, grasping thematic elements, character arcs, or interconnected arguments that span across hundreds or thousands of words.

Consider a scenario like answering a question about the overarching themes of a novel or understanding a complex argument presented across different sections of a technical paper. Retrieving only a few isolated paragraphs, even if individually relevant to certain keywords, may fail to provide the LLM with the necessary context to synthesize information scattered throughout the document. This limitation hinders the model's ability to capture large-scale discourse structure and perform multi-step reasoning that relies on integrating knowledge across lengthy texts. Existing methods based on contiguous segmentation may not capture the complete semantic depth or relationships between distant parts of a document. Reading isolated snippets from technical or scientific documents can even lead to a loss of important context, potentially making the information difficult to interpret or even misleading.

Introducing RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

To overcome the limitations of traditional retrieval based on contiguous chunks, the RAPTOR model introduces a novel approach that structures document knowledge hierarchically using a tree. This method, Recursive Abstractive Processing For Tree-Organized Retrieval, aims to capture both granular details and high-level thematic information, allowing for more effective retrieval and understanding of long texts.

The core idea behind RAPTOR is to build a multi-level representation of a document that moves from fine-grained details at the bottom to broad summaries at the top. This is achieved through a recursive process involving embedding, clustering, and summarization.

How RAPTOR Constructs the Knowledge Tree

The construction of the RAPTOR tree is a bottom-up process:

1. Initial Chunking: The process begins by segmenting the original long document into small, manageable text chunks. These chunks form the leaf nodes at the bottom layer of the tree.
2. Embedding: Each of these initial text chunks is embedded into a dense vector space using a chosen text embedding model. These embeddings capture the semantic meaning of each chunk.
3. Clustering: The embeddings of adjacent nodes (initially the text chunks) are clustered together based on their semantic similarity. This grouping identifies chunks that are conceptually related, even if they are not strictly contiguous in the original text (though initial clustering might favor adjacency).
4. Summarization: For each identified cluster of nodes, an abstractive summary is generated. This summarization step is typically performed by a separate language model, which reads the text content of all nodes within a cluster and generates a concise, high-level summary that captures the main points or themes of that group.
5. Creation of Parent Nodes: Each generated summary becomes the content of a new node in the layer above. These new nodes represent a higher level of abstraction than the nodes they summarize. They also store pointers to their child nodes (the chunks/summaries from the layer below that were clustered and summarized).
6. Recursion: Steps 2-5 are repeated recursively. The newly created summary nodes in the upper layer are treated as the input for the next iteration. Their text content (the summaries) is embedded, these embeddings are clustered, and the resulting clusters are summarized to create nodes for the layer above that. This process continues until a single root node is created, representing a summary of the entire document at its highest level of abstraction.

This recursive process results in a tree structure where the leaf nodes contain the original text chunks, and nodes at progressively higher levels contain summaries that abstract information from their child nodes. Nodes at intermediate levels provide summaries of sections or clusters of ideas, while the root node offers an overview of the entire document. Crucially, this structure explicitly captures hierarchical relationships and allows information to be organized and accessed at different levels of detail.

Enhanced Retrieval During Inference

The true power of the RAPTOR tree structure is realized during the retrieval phase when a user poses a query. Unlike traditional methods that only retrieve individual text chunks, RAPTOR can leverage the multi-level hierarchy.

When a query is received, the system can query the tree to find relevant nodes. Retrieval can occur at any level of the tree, or even across multiple levels. For instance, a query might be relevant to specific details found in the leaf nodes, a broader theme summarized in an intermediate node, or the overall topic captured by the root node.

The retrieval mechanism selects nodes whose content (original text or summaries) is most relevant to the query. By potentially retrieving nodes from different levels, the LLM is provided with a richer, more comprehensive context that includes both specific facts and the higher-level ideas or sections they belong to. This allows the LLM to synthesize information more effectively, understand the broader context, and perform reasoning that requires connecting concepts across different parts of the original document. For example, the LLM can receive both a granular detail about a character from a leaf node and a summary of the character's arc from an intermediate node, providing a much deeper understanding than the detail alone.

Key Contributions and Experimental Evidence

The RAPTOR paper highlights several key contributions:

1. Novel Hierarchical Indexing: The introduction of a recursive process using embedding, clustering, and summarization to build a hierarchical tree representation of long documents for retrieval purposes.
2. Multi-Level Context Provision: Demonstrating that retrieving from different levels of this tree structure provides superior context to LLMs compared to retrieving only contiguous chunks.
3. Experimental Validation: Providing controlled experiments using various language models (UnifiedQA, GPT-3, and GPT-4) that show significant improvements in retrieval-augmented performance when using RAPTOR on collections of long documents.
4. State-of-the-Art Results: Achieving new state-of-the-art results on several challenging question-answering tasks that specifically require processing long texts and complex reasoning. Examples include:
   • NarrativeQA: Free text response questions on books and movies.
   • QASPER: Questions based on full-text NLP research papers.
   • QuALITY: Multiple-choice questions based on medium-length passages, often requiring inference and synthesis across the text.

Specifically, coupling RAPTOR retrieval with GPT-4 demonstrated a significant improvement, such as increasing the best reported performance on the QuALITY benchmark by 20% in absolute accuracy. This result underscores the effectiveness of providing LLMs with context that better reflects the structure and interconnectedness of information within long documents. Even with less powerful models like UnifiedQA, RAPTOR showed performance gains, indicating the method's general applicability.

Comparison with Existing Techniques

The paper positions RAPTOR within the landscape of retrieval-augmented models and summarization techniques. While advances in hardware have increased the maximum context length LLMs can handle, models often struggle to effectively utilize very long contexts, and processing them remains computationally expensive and slow. This reinforces the need for intelligent information selection through retrieval.

Existing retrieval methods predominantly rely on contiguous chunking. Some related work in recursive summarization or hierarchical representation exists, such as approaches that summarize adjacent text chunks (like LlamaIndex). However, these methods often rely heavily on textual adjacency for grouping, potentially missing relationships between distant but semantically connected parts of a document. By using embedding and clustering before summarization, RAPTOR can group semantically similar content regardless of its original position in the text, potentially capturing interdependencies that adjacency-based methods would overlook. The recursive summarization approach allows RAPTOR to compress information losslessly across layers, while the ability to retrieve from any node preserves access to granular details when needed, mitigating potential information loss that can occur in methods relying solely on top-level summaries.

The hierarchical, tree-based structure, built through recursive clustering and summarization, is the key differentiator for RAPTOR, enabling a more sophisticated representation and retrieval strategy for long and complex texts.

Conclusion

RAPTOR presents a significant step forward in retrieval-augmented language models by addressing the challenge of effectively utilizing long document context. Its novel method of building a recursive, tree-organized knowledge representation through embedding, clustering, and summarization allows LLMs to access information at varying levels of abstraction, from fine-grained details to high-level summaries.

The experimental results demonstrate that this hierarchical approach yields substantial performance improvements on tasks requiring deep understanding and integration of information from lengthy texts, achieving state-of-the-art results on prominent benchmarks. By providing LLMs with a more structured and contextually rich representation of source documents, RAPTOR enhances their ability to perform complex reasoning and generate more accurate and comprehensive responses. This research highlights the potential of advanced indexing and retrieval strategies to unlock the full capabilities of large language models when interacting with large and complex bodies of text.

Source(s)

• RAPTOR: RECURSIVE ABSTRACTIVE PROCESSING FOR TREE-ORGANIZED RETRIEVAL Paper

Enjoyed this post? Found it insightful? Feel free to leave a comment below to share your thoughts or ask questions. A GitHub account is required to join the discussion.