Keywords
Large Language Models, Reasoning Evaluation

Technologies
Python, Mistral API, ROSCOE Evaluation

Large language models perform strongly across many tasks, yet their reasoning abilities remains largely unevaluated. As deployment in decision critical settings grows, evaluations still prioritise answer accuracy over the validity and coherence of reasoning. This project introduces a reasoning benchmark and a dual evaluation framework measuring both answer correctness and reasoning quality.

Eighty open-ended questions were developed across four subcategories: number series, arithmetic word problems, verbal analogies, and syllogisms. To limit memorisation, items were adapted from aptitude tests with varied values, wording, and patterns. Three Ministral models were evaluated under answer-only and chain-of-thought prompting, with reasoning quality assessed using the ROSCOE framework.

CoT prompting improved answer accuracy across all models, though gains varied by difficulty and category. ROSCOE results showed reasoning was generally grounded and informative but often redundant with weak step-level coherence. These findings indicate that CoT driven accuracy gains do not fully reflect reasoning quality, highlighting the need to evaluate reasoning processes alongside final answers.