RAGCheck Evaluating Multimodal Retrieval Augmented Generation Performance

This post delves into the RAGCheck framework, a novel approach for evaluating the performance of multimodal Retrieval Augmented Generation (RAG) systems. Introduced in the paper "RAGCheck: Evaluating Multimodal Retrieval Augmented Generation Performance" by Mortaheb et al. (2025), this framework aims to enhance the reliability of Large Language Models (LLMs) by addressing the issue of hallucinations—incorrect or irrelevant information generated by these systems.

Introduction to RAGCheck Framework

The RAGCheck framework introduces two novel metrics to assess the performance of multimodal RAG systems:

1. Relevancy Score (RS): Measures the pertinence of retrieved data (both text and images) to the user's query.
2. Correctness Score (CS): Evaluates the accuracy of the generated response in relation to the retrieved data.

These metrics are designed to address the multifaceted nature of hallucinations, which can arise from the LLM's response generation, the retrieval process itself, and the conversion of multimodal data into text by Vision-Language Models (VLMs).

Training and Validation of RAGCheck Metrics

The authors trained machine learning models for both RS and CS using a dataset derived from ChatGPT and human evaluations. These models achieved high accuracy, around 88%, on test data. Further validation was conducted using a separate human-annotated dataset of 5,000 samples, which demonstrated that the RS model outperformed CLIP in retrieval alignment with human judgment.

Evaluating Correctness Score (CS)

The CS evaluation involves segmenting the generated response into spans, classifying them as objective or subjective, and then scoring the accuracy of the objective spans against the original retrieved data (raw context). This approach ensures that the generated responses are not only relevant but also accurate, thereby reducing hallucinations.

Conclusion

The RAGCheck framework provides a robust method for evaluating the performance of multimodal RAG systems. By focusing on both the relevance of retrieved information and the correctness of generated responses, this framework addresses the challenges posed by hallucinations and offers a valuable tool for improving the reliability of these systems. The authors demonstrate the effectiveness of their approach through empirical results and comparisons with existing methods.

Source(s):

• RAGCheck: Evaluating Multimodal Retrieval Augmented Generation Performance