Zero-RAG: towards retrieval-augmented generation with zero redundant knowledge

Abstract

Retrieval-Augmented Generation (RAG) has shown remarkable results to address the hallucinations of Large Language Models (LLMs) , which usually uses a large external corpus to supplement knowledge. However, with the development of LLMs, the internal knowledge of LLMs has expanded significantly, thus causing significant knowledge redundancy between the external corpus and LLMs. On the one hand, the indexing cost of dense retrieval is highly related to the corpus size and thus significant redundant knowledge intensifies the dense retrieval’s workload. On the other hand, the redundant knowledge in the external corpus is not helpful to LLMs and our exploratory analysis shows that it instead hurts the RAG performance on those questions which the LLM can answer by itself. To address these issues, Zero-RAG is proposed to tackle these challenges. Specifically, a Mastery-Score metric is introduced to identify and prune redundant knowledge in the RAG corpus. After pruning, answers to “mastered” questions rely primarily on internal knowledge of the LLM. To better exploit this capacity, a Query Router and a Noise-Tolerant Tuning strategy are employed, thereby mitigating the distraction of irrelevant documents and enhancing the utilisation of internal knowledge under the pruned corpus. Experimental results show that Zero-RAG prunes the Wikipedia corpus by 30% and accelerates the retrieval stage by 27%, without compromising RAG’s performance (average degradation <2 points).

Graphical abstract

Keywords

large language models (LLMs) / retrieval-augmented generation (RAG) / corpus pruning / retrieval efficiency

References

[1] Lin S, Hilton J, Evans O. TruthfulQA: Measuring how models mimic human falsehoods. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022, 3214−3252

[2] Wang C, Liu X, Yue Y, Tang X, Zhang T, Cheng J, Yao Y, Gao W, Hu X, Qi Z, Wang Y, Yang L, Wang J, Xie X, Zhang Z, Zhang Y. Survey on factuality in large language models: knowledge, retrieval and domain-specificity. 2023, arXiv preprint arXiv: 2310.08521

[3] Gao Y, Xiong Y, Gao X, Jia K, Pan J, Bi Y, Dai Y, Sun J, Wang M, Wang H. Retrieval-augmented generation for large language models: a survey. 2024, arXiv preprint arXiv: 2312.10997

[4] Fan W, Ding Y, Ning L, Wang S, Li H, Yin D, Chua T S, Li Q. A survey on RAG meeting LLMs: towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024, 6491−6501

[5] Li X, Zhu C, Li L, Yin Z, Sun T, Qiu X. LLatrieval: LLM-verified retrieval for verifiable generation. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 5453−5471

[6] Yu H, Gan A, Zhang K, Tong S, Liu Q, Liu Z. Evaluation of retrieval-augmented generation: a survey. 2024, arXiv preprint arXiv: 2405.07437

[7] Asai A, Wu Z, Wang Y, Sil A, Hajishirzi H. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[8] Borgeaud S, Mensch A, Hoffmann J, Cai T, Rutherford E, Millican K, van den Driessche G, Lespiau J B, Damoc B, Clark A, de Las Casas D, Guy A, Menick J, Ring R, Hennigan T, Huang S, Maggiore L, Jones C, Cassirer A, Brock A, Paganini M, Irving G, Vinyals O, Osindero S, Simonyan K, Rae J W, Elsen E, Sifre L. Improving language models by retrieving from trillions of tokens. In: Proceedings of the 39th International Conference on Machine Learning. 2022, 2206−2240

[9] Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A et al, et al. The llama 3 herd of models. 2024, arXiv preprint arXiv: 2407.21783

[10] Guo D, Yang D, Zhang H, Song J, Zhang R et al, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. 2025, arXiv preprint arXiv: 2501.12948

[11] Anil R, Borgeaud S, Alayrac J B, Yu J, Soricut R et al, et al. Gemini: a family of highly capable multimodal models. 2024, arXiv preprint arXiv: 2312.11805

[12] Xiao C, Cai J, Zhao W, Zeng G, Lin B, Zhou J, Zheng Z, Han X, Liu Z, Sun M. Densing law of LLMs. 2024, arXiv preprint arXiv: 2412.04315

[13] Fang L, Wang Y, Liu Z, Zhang C, Jegelka S, Gao J, Ding B, Wang Y. What is wrong with perplexity for long-context language modeling? In: Proceedings of the 13th International Conference on Learning Representations. 2025

[14] Chen Y, Qian S, Tang H, Lai X, Liu Z, Han S, Jia J. LongLoRA: Efficient fine-tuning of long-context large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[15] Ding J, Ma S, Dong L, Zhang X, Huang S, Wang W, Zheng N, Wei F. LongNet: Scaling transformers to 1,000,000,000 tokens. 2023, arXiv preprint arXiv: 2307.02486

[16] Peng B, Quesnelle J, Fan H, Shippole E. YaRN: Efficient context window extension of large language models. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[17] Allen-Zhu Z, Li Y. Physics of language models: Part 3.1, knowledge storage and extraction. In: Proceedings of the 41st International Conference on Machine Learning. 2024, 1067−1077

[18] Sciavolino C, Zhong Z, Lee J, Chen D. Simple entity-centric questions challenge dense retrievers. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing. 2021, 6138−6148

[19] Joshi M, Choi E, Weld D S, Zettlemoyer L. TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017, 1601−1611

[20] Wang M, Sunkara S, Baechler G, Lin J, Zhu Y, Zubach F, Shu L, Chen J. WebQuest: a benchmark for multimodal QA on web page sequences. 2024, arXiv preprint arXiv: 2409.13711

[21] Yang Z, Qi P, Zhang S, Bengio Y, Cohen W W, Salakhutdinov R, Manning C D. HotpotQA: a dataset for diverse, explainable multi-hop question answering. In: Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing. 2018, 2369−2380

[22] Zhu D, Wang L, Yang N, Song Y, Wu W, Wei F, Li S. LongEmbed: Extending embedding models for long context retrieval. In: Proceedings of 2024 Conference on Empirical Methods in Natural Language Processing. 2024, 802−816

[23] Mao K, Deng C, Chen H, Mo F, Liu Z, Sakai T, Dou Z. ChatRetriever: adapting large language models for generalized and robust conversational dense retrieval. In: Proceedings of 2024 Conference on Empirical Methods in Natural Language Processing. 2024, 1227−1240

[24] Zhang T, Patil S G, Jain N, Shen S, Zaharia M, Stoica I, Gonzalez J E. RAFT: adapting language model to domain specific RAG. 2024, arXiv preprint arXiv: 2403.10131

[25] Yu W, Zhang H, Pan X, Cao P, Ma K, Li J, Wang H, Yu D. Chain-of-note: Enhancing robustness in retrieval-augmented language models. In: Proceedings of 2024 Conference on Empirical Methods in Natural Language Processing. 2024, 14672−14685

[26] Yao Z, Qi W, Pan L, Cao S, Hu L, Liu W, Hou L, Li J. SeaKR: Self-aware knowledge retrieval for adaptive retrieval augmented generation. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. 2025, 27022−27043

[27] Fei W, Niu X, Zhou P, Hou L, Bai B, Deng L, Han W. Extending context window of large language models via semantic compression. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024, 5169−5181

[28] Chevalier A, Wettig A, Ajith A, Chen D. Adapting language models to compress contexts. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 3829−3846

[29] Ge T, Hu J, Wang L, Wang X, Chen S Q, Wei F. In-context autoencoder for context compression in a large language model. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[30] Xu F, Shi W, Choi E. RECOMP: improving retrieval-augmented LMs with context compression and selective augmentation. In: Proceedings of the 12th International Conference on Learning Representations. 2024

[31] Zhang Z, Zhu A, Yang L, Xu Y, Li L, Phothilimthana P M, Jia Z. Accelerating iterative retrieval-augmented language model serving with speculation. In: Proceedings of the 41st International Conference on Machine Learning. 2024, 60626−60643

[32] Faysse M, Sibille H, Wu T, Omrani B, Viaud G, Hudelot C, Colombo P. ColPali: efficient document retrieval with vision language models. In: Proceedings of the 13th International Conference on Learning Representations. 2025

[33] Jiang Z, Xu F, Gao L, Sun Z, Liu Q, Dwivedi-Yu J, Yang Y, Callan J, Neubig G. Active retrieval augmented generation. In: Proceedings of 2023 Conference on Empirical Methods in Natural Language Processing. 2023, 7969−7992

[34] Liu Y, Peng X, Zhang X, Liu W, Yin J, Cao J, Du T. RA-ISF: Learning to answer and understand from retrieval augmentation via iterative self-feedback. In: Proceedings of the Findings of the Association for Computational Linguistics. 2024, 4730−4749

[35] Jeong S, Baek J, Cho S, Hwang S J, Park J. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In: Proceedings of 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2024, 7036−7050

[36] Wang F, Wan X, Sun R, Chen J, Arik S O. Astute RAG: overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025, 30553−30571

[37] Chen J, Lin H, Han X, Sun L. Benchmarking large language models in retrieval-augmented generation. In: Proceedings of the 38th AAAI Conference on Artificial Intelligence. 2024, 17754−17762

RIGHTS & PERMISSIONS

Higher Education Press