Abstract: Retrieval-Augmented Generation (RAG) systems have become the default way we deploy LLMs over proprietary knowledge, yet the way we test them has not kept pace. Most existing frameworks score answers one query at a time against a fixed snapshot of the corpus, which is useful for benchmarking, but a long way from testing the system as a whole. In this talk I want to take a software-testing view of RAG systems and ask two questions:
First, is our test suite actually exercising the retriever, or are we just hitting the same popular chunks over and over? I’ll introduce Chunk Coverage, an oracle-independent adequacy criterion. Second, what happens when the corpus moves under your feet: updates, stale versions, OCR noise, format drift? I’ll talk about metamorphic testing for RAG.


Chair: Carmen Armenti