Keywords
Irish, LLM, NLP, benchmarks, leaderboard

Technologies
Python, Gradio, Docker, Pandas, APIs

Large language models (LLMs) are increasingly embedded in everyday technology, yet their evaluation remains overwhelmingly English-centric. Low-resource languages such as Irish are routinely excluded from mainstream benchmarking, leaving a critical gap in our understanding of how modern AI systems perform in these linguistic contexts.

This project presents the Irish LLM Leaderboard, a web-based platform that evaluates and ranks large language models across five Irish-focused benchmarks: IRLBench, a Leaving Certificate-derived reasoning dataset; IrishQA, a multiple-choice question answering dataset; Irish-BLiMP, a linguistic minimal pairs benchmark; IrishMultiJail, a safety evaluation dataset; and Irish GDPval, a general language proficiency benchmark.

The platform is built using Python and Gradio, deployed via Docker, and draws live data from a dynamically updated source. It gives researchers and developers a transparent, reproducible, and openly accessible tool for assessing LLM capability in Irish supporting the broader goal of ensuring AI development does not leave minority languages behind.