Attacking AI

(Live April 17-18th)

(Note: the course will be recorded and distributed to students after completion.)

This two-day course provides security professionals with hands-on experience in assessing and attacking AI systems. Participants will explore real-world vulnerabilities in LLMs and AI-integrated applications. Through practical labs and case studies, students will develop methodologies to conduct AI security assessments and implement defensive strategies.

Course Details

• Format: Hybrid of lectures, interactive discussions, and hands-on labs
• Prerequisites: Intermediate cybersecurity knowledge and a basic understanding of AI concepts
• Class Collaboration: Participants will have access to private Discord channels shared with the "Red Blue Purple AI" course for discussion and resource sharing.

Note: This syllabus is subject to updates, as AI security is a rapidly evolving field.

Day 1: AI Attack Surfaces and Offensive Techniques

The AI Gold Rush

• Rapid adoption of AI and its security implications
• Key industries integrating AI and associated risks
• Traditional security vulnerabilities in AI applications

Common AI Architectures and Ecosystems

• AI development pipeline: model selection, training, and deployment
• Infrastructure components: cloud services, APIs, and agentic architectures
• The role of LLMs, RAG (Retrieval-Augmented Generation), and AI agents in modern systems

Understanding AI Threat Modeling

• Common AI security threat models
• Threat modeling LLM-based systems
• Threat modeling image-based AI systems
• Practical exercise: Threat modeling a real-world AI deployment

Introduction to Prompt Injection

(Early in Day 1 – Foundational Concepts & Hands-on Exercises)

• What is Prompt Injection?
• Types: Fuzzing (gradient) vs Logical
• Understanding LLM prompt processing and model constraints
• Prompt Injection Techniques (Introduction)
• Basic prompt manipulation methods
• Early case studies of prompt injection in real-world applications
• Hands-on Lab: Crafting basic prompt injection attacks

LLM Jailbreaking for Security Professionals

• Common jailbreak techniques
• Review Case Studies

Evening Wrap-Up & Discussion

• Open Q&A session on the day's topics

Day 2: Attacking AI In-Depth and Defensive Methodologies

AI Red Teaming Methodologies

• Industry approaches to AI red teaming
• Key players in AI security testing

Attacking AI-Integrated Applications

• AI-powered APIs and their security risks
• API security for LLM-based systems

MITRE ATLAS & OWASP AI Top Ten

• Overview of MITRE’s ATLAS framework
• OWASP Top 10 vulnerabilities for LLM systems
• Mapping AI threats to traditional security vulnerabilities

The Arcanum LLM Assessment Methodology

• A structured methodology for assessing AI system security
• Checklist and guidance for AI penetration testers
• Case study: Security auditing an enterprise AI deployment

Arcanum’s Prompt Injection Taxonomy

(This is the advanced section, distinct from Day 1’s introduction)

Prompt Injection Primitives

• Attack Intents (Manipulating LLM objectives - 10+ areas of focus)
• Attack Techniques (Tools to execute your objectives. 15+ areas of focus)
• Evasion Methods (Methods to bypass input and output "filters". 27+ areas of focus)
• Core techniques that can be adapted across multiple AI models
• Real-world examples demonstrating effectiveness

Defending AI Systems

The Arcanum AI Defense Layers

• Layer One (Ecosystem): Securing AI infrastructure and cloud environments
• Layer Two (Model): Protecting AI models from poisoning and adversarial attacks
• Layer Three (Prompt): Preventing prompt injection and response manipulation
• Layer Four (Data): Safeguarding training and inference data from corruption
• Layer Five (Application): Hardening AI-integrated applications and APIs

Wrap-Up Discussion and Q&A

• Final thoughts on AI security trends
• Emerging threats and future considerations
• Open discussion and networking