Keywords
Agentic, LLMs, Benchmark, Pathfinding, Evaluation

Technologies
LangGraph, LangChain, Python, Claude, OpenAI

Recent developments in Agentic AI allow large language models to act as orchestrators, solving complex tasks through multi-step reasoning and tool use. Most existing benchmarking solutions focus on final outcomes, overlooking the reliability of decisions at each step. This project introduces a benchmarking framework to compare LLMs on their reliability and performance as autonomous orchestrators. We evaluate multiple LLMs on grid-based pathfinding tasks, where LLMs must compute shortest paths using tools and autonomous reasoning, with Dijkstra’s algorithm as ground truth. The benchmark offers various task complexity, prompting strategies, and levels of autonomy, including configurations with and without validation tools. Reliability is measured through path correctness, optimality, tool usage, runtime, and hallucination rates with increasing graph sizes. By isolating a single-LLM architecture, this work provides a clearer analysis of orchestration capabilities and failures, offering insights into how well LLMs can plan, execute, and backtrack in structured, controlled environments.