Comparative benchmarking of retrieval-augmented generation reranker for medical domain

Abstract

The exponential growth of digital medical information poses significant challenges in delivering reliable, evidence-based responses to clinical inquiries. Traditional systems often fall short in bridging the gap between vast data repositories and the need for authoritative, contextually relevant insights. In this study, we introduce a pipeline that leverages a Retrieval-Augmented Generation reranker architecture, combined with a Chain-of-Thought (CoT) prompting strategy, to enhance the performance of Large Language Models in addressing complex medical questions. By integrating a robust retrieval mechanism that sources trustworthy evidence from established medical literature and by refining the information with reranking, our approach not only improves answer accuracy but also demonstrates that larger models can be effectively distilled into smaller, more resource-efficient variants while maintaining comparable performance. The pipeline is evaluated in zero-shot question-answering scenarios, employing a question-only retrieval strategy to simulate realistic clinical contexts where prior domain-specific fine-tuning is absent. This work underscores the potential of combining retrieval techniques with sequential reasoning to overcome the inherent challenges in medical AI, paving the way for more accurate, transparent, and accessible systems in healthcare applications.