Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation
AI Summary
This paper benchmarks seven models for AWS Terraform generation across 17 scenarios: Claude Opus 4, GPT-5.4, Gemini 2.5 Pro, Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, and Magicoder-S-CL-7B. The study combines Checkov and Trivy with a GitLab CI/CD pipeline and evaluates three security prompt levels using pass@5. Its central finding is that syntactic validity and security compliance are largely orthogonal. WizardCoder-33B reaches a 77.8% validation rate but zero Checkov compliance, while Claude Opus 4 reaches 23.1% Checkov and 92.5% Trivy pass rates with detailed security prompting. The authors conclude that automated multi-tool scanning remains necessary regardless of model family or prompt strategy.
Why it's worth reading
As generated infrastructure moves closer to deployment, this benchmark provides concrete evidence that valid Terraform is not necessarily secure Terraform, with direct implications for IaC evaluation, prompting, and CI security gates.
Deep Read
1. What happened
Original facts: The paper evaluates security in text-to-Terraform generation across seven models, 17 AWS scenarios, three security prompt levels, and pass@5. Checkov and Trivy are integrated into a GitLab CI/CD pipeline, and the abstract states that all artifacts are publicly available.
2. Core technology
Original facts: The models are Claude Opus 4, GPT-5.4, Gemini 2.5 Pro, Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, and Magicoder-S-CL-7B. The evaluation separates Terraform validation from Checkov security compliance and Trivy pass results.
3. Key evidence and numbers
Original facts: WizardCoder-33B achieves a 77.8% validation rate but zero Checkov compliance. Under detailed security prompting, Claude Opus 4 reaches 23.1% Checkov and 92.5% Trivy pass rates. The abstract does not include per-scenario results, confidence intervals, or the complete comparison table.
4. Why it matters
Analysis: The reported gap demonstrates that parseable or valid infrastructure code is not equivalent to secure infrastructure code. A single syntax or validation metric can therefore understate cloud-configuration risk, while multiple scanners provide a more operationally relevant DevSecOps control.
5. Practical impact
Analysis: Engineering teams can treat model-generated Terraform as untrusted input and require Checkov, Trivy, or comparable checks before merge and deployment. Evaluation pipelines should report syntax validity, policy violations, and vulnerability findings separately. Prompt engineering should complement automated controls rather than replace them.
6. Limitations and uncertainty
Original facts and analysis: The abstract does not specify the scenario composition, total sample count, randomness controls, scanner versions, rule sets, false-positive handling, or model cost and latency. Reproducibility and generalizability therefore require checking the full paper and released artifacts. Checkov and Trivy cover different risk surfaces, so their pass rates should not be interpreted as a single probability of overall security. The abstract provides no evidence about cross-cloud transfer or reductions in real production incidents.
7. Original sources
- arXiv abstract and paper record
- Paper: Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation
- arXiv ID: 2608.02672
- Publication date: 2026-08-02