Keywords
LLM Agent, Constraint Satisfaction, Scheduling

Technologies
LangGraph, Ollama, Choco Solver, Google Calendar

This project investigates the performance of large language models (LLMs) in deterministic problem-solving. The problem type is constraint-based calendar scheduling. LLMs have strong abilities in natural language interpretation and reasoning, but they are still probabilistic in nature as they generate text based on likely next tokens. This means that they can struggle with complex tasks requiring reliable decision making or strict constraint satisfaction.

To evaluate LLM performance in scheduling tasks, an LLM agent was developed that can understand users' scheduling requests, break them down into constraint satisfaction problems, gather any additional information that may be required, and update relevant Google Calendars. The system also includes a constraint solving program that must verify all scheduling solutions before any action is taken to implement them.

Using this system, tests were carried out with increasingly difficult problems on a variety of LLMs. These evaluated the problem-solving capabilities of each model and explored the increase in performance with schedule verification. The project's resulted in a functional AI-driven Google Calendar management system.