Keywords
LLMs, Quantization, Benchmark, AI, ML

Technologies
llama-cpp, FAISS, Python, scikit-learn, CUDA

LLMs have demonstrated remarkable capabilities in natural language understanding, but their substantial computational requirements restrict deployment to those with specialised infrastructure. Quantization addresses this by reducing the numerical precision of model weights, enabling deployment on consumer hardware, but reducing accuracy. RAG offers a potential remedy by supplying external knowledge at inference time, reducing reliance on parametric memory. This project investigates whether RAG can compensate for accuracy loss caused by aggressive two-bit post-training quantization.

An evaluation pipeline supporting both lexical (exact) and semantic (meaning) retrieval was evaluated against the AI2 ARC benchmark (1,172 questions). A no-RAG baseline established a 5.29pp gap between FP16 and Q2_K models. Neither retrieval strategy improved upon this baseline; however, FAISS RAG narrowed the gap to 4.87 points while TF-IDF widened it to 6.57 points. Results revealed that quantized models are disproportionately sensitive to retrieval quality, suffering greater accuracy loss than their full-precision counterparts when retrieved context is noisy or irrelevant.