Garak vs EU AI Act: Mapping LLM Testing to Compliance

What Garak is, and why NVIDIA built it

Garak is NVIDIA's open-source LLM vulnerability scanner. If you know nmap for network security or Metasploit for penetration testing, Garak does the equivalent job for large language models. It probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and a growing list of other failure modes. The project is actively maintained on GitHub with regular releases, a published user guide at docs.garak.ai, and community adoption that makes it the de facto open-source standard for LLM security testing.

Garak supports essentially every LLM interface that matters: Hugging Face Hub, OpenAI API, AWS Bedrock, LiteLLM, Replicate, GGUF models, and anything accessible via REST. It runs from the command line. It takes a model name and runs its full suite of probes, or you can target specific vulnerability classes. The output is structured JSON you can integrate into CI/CD pipelines or compliance workflows.

But here is what Garak's documentation does not say anywhere: the words "compliance," "regulation," "EU AI Act," "ISO," and "NIST" do not appear in a single page of the docs. Garak is a security testing tool built by security researchers. It was not designed as a regulatory compliance solution. That matters, because enterprises are increasingly reaching for open-source testing tools to satisfy the cybersecurity testing obligations in the EU AI Act. Understanding where Garak aligns with regulatory requirements, and where it does not, is the difference between due diligence and a compliance gap you do not know you have.

What Garak actually tests

Garak's probes fall into several distinct categories that map roughly to known LLM failure modes:

• Hallucination probes: Test whether the model generates factually incorrect or fabricated outputs when prompted with questions that have verifiable answers. These probes check for confabulation in domains like history, science, and current events.
• Data leakage probes: Attempt to extract training data from the model, including personally identifiable information, copyrighted text, and memorised training examples. These are confidentiality-focused tests.
• Prompt injection probes: Test whether the model can be made to ignore its system instructions or safety guardrails through carefully crafted inputs. This includes direct injection, indirect injection via retrieved documents, and multi-turn injection attacks.
• Jailbreak probes: Assess whether the model's refusal mechanisms can be bypassed to produce harmful content, including instructions for illegal activities, hate speech, or dangerous information.
• Toxicity generation probes: Measure whether the model produces toxic, offensive, or discriminatory outputs across a range of identity dimensions.
• Misinformation probes: Test the model's susceptibility to generating or amplifying false narratives, conspiracy content, and misleading claims.

Garak combines static, dynamic, and adaptive probing strategies. Static probes send fixed payloads. Dynamic probes generate test cases algorithmically. Adaptive probes modify their approach based on the model's responses. This three-layer approach is more sophisticated than most commercial testing suites, and it is one reason Garak has gained traction in security-conscious organisations.

What EU AI Act Article 15 actually demands

The EU AI Act's cybersecurity requirements for high-risk AI systems sit in Article 15(5). The text is specific about what kinds of vulnerabilities must be addressed:

"The technical solutions to address AI specific vulnerabilities shall include, where appropriate, measures to prevent, detect, respond to, resolve and control for attacks trying to manipulate the training data set (data poisoning), or pre-trained components used in training (model poisoning), inputs designed to cause the AI model to make a mistake (adversarial examples or model evasion), confidentiality attacks or model flaws."

This is not a general "test your model for security issues" requirement. It is a specific list of five vulnerability classes that the regulation explicitly names: data poisoning, model poisoning, adversarial examples/model evasion, confidentiality attacks, and model flaws. The regulation also requires measures to prevent, detect, respond to, resolve, and control for these attacks. A testing tool that runs probes once during development satisfies only the "detect" function. Prevention, response, resolution, and ongoing control require operational processes, monitoring infrastructure, and incident response procedures that no scanner provides on its own.

Article 15(4) also requires resilience against errors, faults, and inconsistencies, including feedback loops in continuously learning systems. This is a robustness requirement that goes beyond security testing into system architecture and operational design.

Mapping Garak's probes to Article 15(5) requirements

Here is where the rubber meets the road. Let us map Garak's probe categories against each of the five vulnerability classes the EU AI Act names:

Data poisoning (Article 15(5)): Garak does not directly test for data poisoning. Data poisoning involves manipulating training data before the model is trained. Garak tests the trained model's outputs, not the provenance or integrity of its training data. To satisfy this requirement, you would need data pipeline validation, training data provenance tracking, and statistical outlier detection in training datasets. Garak cannot tell you whether your training data was poisoned.

Model poisoning (Article 15(5)): Similar gap. Model poisoning targets pre-trained components or weights. Garak tests the model's behaviour at inference time. It does not verify model weight integrity, check for backdoors embedded during fine-tuning, or validate the supply chain of pre-trained components. Supply chain attestation, model signing, and weight verification are separate concerns.

Adversarial examples / model evasion (Article 15(5)): This is where Garak provides the strongest alignment. Garak's prompt injection probes, jailbreak probes, and adaptive testing strategies are direct tests for model evasion. An evasion attack is exactly what prompt injection and jailbreaking attempt: inputs designed to cause the model to make a mistake by bypassing its intended behaviour. Garak's multi-turn and indirect injection probes go further than most tools by testing evasion across different attack surfaces. This is genuine regulatory coverage.

Confidentiality attacks (Article 15(5)): Garak's data leakage probes provide meaningful coverage here. Extracting training data, PII, or proprietary information from a model is a confidentiality attack. Garak specifically tests for memorised training data extraction and PII leakage. However, Garak does not test for model inversion attacks (reconstructing training data from model outputs) or membership inference attacks (determining whether a specific data point was in the training set). These are confidentiality attack vectors that are well documented in the security literature but fall outside Garak's current probe set.

Model flaws (Article 15(5)): Garak's hallucination, toxicity, and misinformation probes all test for model flaws. A model that hallucinates factual content, generates toxic output, or amplifies misinformation has functional flaws. Garak provides structured, repeatable testing for these flaw categories. This is genuine coverage, though "model flaws" as a regulatory category is deliberately broad and no single tool can claim exhaustive coverage of every possible flaw in every possible deployment context.

Three gaps you need to fill yourself

1. Accuracy measurement and declaration. Article 15(3) requires that "the levels of accuracy and the relevant accuracy metrics of high-risk AI systems shall be declared in the accompanying instructions of use." Garak does not measure accuracy in the regulatory sense. It tests for failures, not for precision, recall, or domain-specific performance benchmarks. If you deploy a high-risk AI system, you need accuracy metrics. Garak will not give them to you.

2. Lifecycle coverage. Article 15(1) requires consistent performance "throughout their lifecycle." Garak runs at a point in time. It does not monitor models in production, detect drift, or alert on degradation. For lifecycle compliance, you need runtime monitoring tools alongside point-in-time testing tools. Garak's structured JSON output does integrate with CI/CD pipelines, which means you can run it at every deployment. That gets you closer to lifecycle coverage than a one-off manual test, but it is not the same as continuous production monitoring.

3. Prevention and response, not just detection. Article 15(5) uses the chain "prevent, detect, respond to, resolve and control." Garak only helps with detection. Prevention means guardrails, input validation, output filtering, and architectural defences. Response means incident playbooks, rollback procedures, and regulatory reporting under Article 73 (serious incident reporting). Resolution means patching, retraining, or decommissioning affected systems. Control means ongoing governance. No testing tool can do any of this. These are organisational capabilities, and they are what the regulation actually demands.

What buyers should verify

If you are evaluating Garak as part of your EU AI Act compliance programme, ask these questions:

• Does our testing pipeline cover all five vulnerability classes in Article 15(5), or only the ones Garak tests? If the answer is "only the ones Garak tests," you have gaps in data poisoning and model poisoning coverage.
• Do we have a method for declaring accuracy metrics as required by Article 15(3), separate from our security testing tooling?
• Is our testing integrated into a lifecycle approach, or do we run Garak once and call it done?
• Do we have prevention, response, and resolution capabilities beyond what our testing tool provides?
• Where does Garak end and where does our organisational process begin? If you cannot answer this question clearly, your compliance programme has an ownership gap.

Garak is genuinely excellent infrastructure for LLM security testing. It is free, well maintained, actively developed by a major research organisation, and more sophisticated than many commercial alternatives. Deploying it in your CI/CD pipeline is a strong signal that you take AI security seriously. Just do not confuse running Garak with satisfying Article 15. The regulation asks for a lot more than a vulnerability scan.

If you are building your AI governance capability and need help mapping your testing infrastructure to regulatory requirements, get in touch. We help organisations understand what the EU AI Act, ISO 42001, and the NIST AI RMF actually require, and where their tooling covers the gap. See our sample audit report or review our pricing. For more on this topic, read our analysis of the gap between AI governance platforms and actual compliance and our guide to AI vendor security questionnaire gaps.

Written by David Swan, reviewed and fact-checked against primary regulatory sources. AI-assisted but human-directed.

Related reading

Always-On AI Wearables — The Hidden Recording Devices in Your Workplace

Free vs Paid AI Tools: The Compliance Cost Nobody Talks About

Reading Between the Lines of an AI Vendor's DPA