Detecting & Mitigating Prompt Injection Attacks

Abstract

This project will develop a real-time system to detect prompt injection attacks in AI applications. Using a layered pipeline of lightweight filters, machine learning classifiers, and large models as a fallback, it will balance accuracy and efficiency. The tool will be tested, benchmarked, and released as open-source software.
public 10 months ago 9 months ago BScDSA
Engagements
SupervisorSecond ReaderAuthorAssigned to
Andrea Visentin Enkhbold Nyamsuren Andrea Visentin Ishaan Arakkal Anwar

Description https://project.cs.ucc.ie/project/1509

This project will tackle the growing security problem of prompt injection attacks, where hidden instructions are embedded in normal text to manipulate AI systems. Such attacks can appear in unexpected places, like shipping addresses or product descriptions, and pose risks for any application that processes external text through an API.

The aim is to design and implement a real-time detection pipeline that balances speed and accuracy. The system will use quick, lightweight checks for most inputs, more advanced machine learning classifiers when something looks suspicious, and large AI models as a final safeguard for the most complex cases.

The detection tool will be developed as open-source software, tested against existing datasets and newly created adversarial examples, and evaluated for both accuracy and efficiency.