Validating Agent-Written Code Before It Runs

An AI agent writes a Python script. You run it. It fails with an ImportError on a library that doesn't exist. Or worse — it runs, silently hits the wrong API endpoint, and corrupts your data. Agent-generated code has specific failure modes that humans don't cause. This playbook gives you a 5-check validation pipeline that catches them before anything executes, plus a Python script that automates all five checks in under 10 seconds.

The Specific Failure Modes of Agent-Written Code

Human developers know what's in their environment. Agents don't. Here's what breaks in production:

1. Hallucinated imports

The agent imports a library that sounds plausible but doesn't exist. Classic examples:

# Agent hallucinated these — none of them exist on PyPI
import anthropic_tools # not a real package
from openai import ChatStream # not a real class (it's stream=True in the API)
import stripe.checkout_session # wrong import path

2. Wrong method signatures

The library exists, but the agent used a method name or argument from a different version or invented it entirely:

# Actual Anthropic SDK (v0.25+):
client.messages.create(model="...", max_tokens=..., messages=[...])

# What agents frequently write (wrong):
client.complete(prompt="...", max_tokens_to_sample=1000) # old API
client.chat(messages=[...]) # doesn't exist

3. API version mismatches

Agent was trained on docs from 6 months ago. The API changed. The code is syntactically correct but functionally broken.

4. Security anti-patterns

Agents hardcode secrets, use eval(), write to arbitrary paths, or call subprocess.run(shell=True) with unsanitized input. These pass all functional tests and fail catastrophically in production.

5. Silent failures

The most dangerous class: code that runs without errors but does the wrong thing. Wrong field name in an API response, off-by-one in a date calculation, wrong boolean logic in a filter. These require a dry run check with known inputs.

The 5-Check Validation Pipeline

The Validation Script

Save as validate_agent_code.py. Run it on any agent-generated .py file before executing it.

#!/usr/bin/env python3
# validate_agent_code.py
# Usage: python3 validate_agent_code.py path/to/agent_generated.py
# Exit code 0 = pass, 1 = fail

import ast
import sys
import re
import importlib
import subprocess
from pathlib import Path
from dataclasses import dataclass, field

@dataclass
class ValidationResult:
 check: str
 passed: bool
 message: str
 details: list[str] = field(default_factory=list)

def check_syntax(filepath: str) -> ValidationResult:
 """Check 1: Parse with AST — catches all syntax errors."""
 try:
 with open(filepath, "r") as f:
 source = f.read()
 ast.parse(source)
 return ValidationResult("syntax", True, "No syntax errors")
 except SyntaxError as e:
 return ValidationResult(
 "syntax", False,
 f"Syntax error at line {e.lineno}: {e.msg}",
 [f" {e.text}"]
 )

def check_imports(filepath: str) -> ValidationResult:
 """Check 2: Verify all imported packages are installed."""
 with open(filepath, "r") as f:
 source = f.read()

 try:
 tree = ast.parse(source)
 except SyntaxError:
 return ValidationResult("imports", False, "Cannot check imports — syntax error", [])

 # Collect top-level module names from import statements
 modules = set()
 for node in ast.walk(tree):
 if isinstance(node, ast.Import):
 for alias in node.names:
 modules.add(alias.name.split(".")[0])
 elif isinstance(node, ast.ImportFrom):
 if node.module:
 modules.add(node.module.split(".")[0])

 # Skip stdlib modules
 stdlib = {
 "os", "sys", "re", "json", "time", "datetime", "pathlib", "typing",
 "subprocess", "collections", "itertools", "functools", "math",
 "hashlib", "hmac", "base64", "io", "abc", "copy", "dataclasses",
 "enum", "logging", "random", "string", "struct", "threading",
 "urllib", "http", "email", "html", "xml", "csv", "sqlite3",
 "ast", "inspect", "importlib", "textwrap", "shutil", "tempfile",
 "socket", "ssl", "uuid", "contextlib", "weakref", "gc", "__future__"
 }

 missing = []
 for module in sorted(modules - stdlib):
 try:
 importlib.import_module(module)
 except ImportError:
 missing.append(module)

 if missing:
 return ValidationResult(
 "imports", False,
 f"{len(missing)} import(s) not found: {', '.join(missing)}",
 [f" pip install {m}" for m in missing]
 )
 return ValidationResult("imports", True, f"All {len(modules - stdlib)} packages installed")

def check_api_methods(filepath: str) -> ValidationResult:
 """Check 3: Verify methods/attrs exist for known SDKs."""
 with open(filepath, "r") as f:
 source = f.read()

 # Known dangerous patterns: method calls that don't exist in current SDK versions
 KNOWN_ANTIPATTERNS = [
 # Anthropic SDK old API
 (r'\.complete\(', "anthropic", "client.complete() — use client.messages.create()"),
 (r'anthropic\.Client\(', "anthropic", "anthropic.Client() — use anthropic.Anthropic()"),
 # OpenAI old API
 (r'openai\.ChatCompletion\.create', "openai", "ChatCompletion.create — use client.chat.completions.create()"),
 (r'openai\.Completion\.create', "openai", "Completion.create — legacy, use chat completions"),
 # Stripe
 (r'stripe\.Charge\.create', "stripe", "stripe.Charge.create — deprecated, use PaymentIntents"),
 ]

 warnings = []
 for pattern, package, message in KNOWN_ANTIPATTERNS:
 if re.search(pattern, source):
 try:
 importlib.import_module(package) # only warn if package is installed
 warnings.append(f" {message}")
 except ImportError:
 pass # package not installed — import check will catch it

 if warnings:
 return ValidationResult(
 "api_methods", False,
 f"{len(warnings)} deprecated/invalid API pattern(s) found",
 warnings
 )
 return ValidationResult("api_methods", True, "No known API anti-patterns detected")

def check_security(filepath: str) -> ValidationResult:
 """Check 4: Flag security anti-patterns."""
 with open(filepath, "r") as f:
 source = f.read()

 SECURITY_PATTERNS = [
 # Hardcoded secrets (rough heuristic)
 (r'(?:api_key|secret|password|token)\s*=\s*["\'][a-zA-Z0-9_\-]{16,}["\']',
 "Possible hardcoded secret — use environment variables"),
 # Dangerous eval/exec
 (r'\beval\s*\(', "eval() usage — avoid or justify explicitly"),
 (r'\bexec\s*\(', "exec() usage — avoid or justify explicitly"),
 # Unsafe subprocess
 (r'subprocess\.(run|call|check_output)\([^)]*shell\s*=\s*True',
 "subprocess with shell=True — dangerous with unsanitized input"),
 # Arbitrary file writes
 (r'open\([^)]*["\'][wa]["\']', "File write detected — verify path is intentional"),
 # Pickle (remote code execution risk)
 (r'\bpickle\b', "pickle usage — RCE risk if loading untrusted data"),
 ]

 issues = []
 for pattern, message in SECURITY_PATTERNS:
 matches = re.findall(pattern, source, re.IGNORECASE)
 if matches:
 issues.append(f" ⚠ {message}")

 if issues:
 return ValidationResult(
 "security", False,
 f"{len(issues)} security concern(s) found — review before running",
 issues
 )
 return ValidationResult("security", True, "No security anti-patterns detected")

def check_dry_run(filepath: str) -> ValidationResult:
 """Check 5: Import-only dry run — catches runtime import errors and immediate exceptions."""
 result = subprocess.run(
 [sys.executable, "-c", f"import ast, sys; ast.parse(open('{filepath}').read()); print('parse ok')"],
 capture_output=True, text=True, timeout=10
 )
 if result.returncode != 0:
 return ValidationResult("dry_run", False, "Dry run failed", [result.stderr.strip()])

 # Try to compile and check for obvious runtime errors in top-level code
 result2 = subprocess.run(
 [sys.executable, "-m", "py_compile", filepath],
 capture_output=True, text=True, timeout=10
 )
 if result2.returncode != 0:
 return ValidationResult("dry_run", False, "Compile check failed", [result2.stderr.strip()])

 return ValidationResult("dry_run", True, "Dry run passed (syntax + compile check)")

def run_validation(filepath: str) -> bool:
 """Run all 5 checks and print results. Returns True if all pass."""
 print(f"\n🔍 Validating: {filepath}\n{'─' * 50}")

 checks = [
 check_syntax(filepath),
 check_imports(filepath),
 check_api_methods(filepath),
 check_security(filepath),
 check_dry_run(filepath),
 ]

 all_passed = True
 for r in checks:
 icon = "✅" if r.passed else "❌"
 print(f"{icon} [{r.check.upper()}] {r.message}")
 for detail in r.details:
 print(detail)
 if not r.passed:
 all_passed = False

 print(f"\n{'─' * 50}")
 if all_passed:
 print("✅ All checks passed. Safe to execute.")
 else:
 print("❌ Validation failed. Fix issues before running.")
 return all_passed

if __name__ == "__main__":
 if len(sys.argv) < 2:
 print("Usage: python3 validate_agent_code.py ")
 sys.exit(1)

 filepath = sys.argv[1]
 if not Path(filepath).exists():
 print(f"File not found: {filepath}")
 sys.exit(1)

 passed = run_validation(filepath)
 sys.exit(0 if passed else 1)

Integrating Into an Agentic Workflow

The validation script is designed to run as a gate between code generation and code execution. Here's the pattern:

# In your agentic orchestration script:
import subprocess
import tempfile
import os

def generate_and_validate(agent_prompt: str, max_retries: int = 2) -> str:
 """Generate code from agent, validate it, retry on failure."""
 for attempt in range(max_retries + 1):
 # 1. Generate code from agent
 code = call_agent(agent_prompt)

 # 2. Write to temp file
 with tempfile.NamedTemporaryFile(mode="w", suffix=".py", delete=False) as f:
 f.write(code)
 temp_path = f.name

 # 3. Validate
 result = subprocess.run(
 ["python3", "validate_agent_code.py", temp_path],
 capture_output=True, text=True
 )

 if result.returncode == 0:
 # Validation passed — return the code
 os.unlink(temp_path)
 return code
 else:
 # Validation failed
 if attempt < max_retries:
 # Retry with error context injected into prompt
 error_context = result.stdout + result.stderr
 agent_prompt = f"""{agent_prompt}

PREVIOUS ATTEMPT FAILED VALIDATION:
{error_context}

Fix these issues in your next attempt. Output only the corrected Python code."""
 os.unlink(temp_path)
 else:
 # Max retries hit — escalate
 os.unlink(temp_path)
 raise ValueError(f"Code validation failed after {max_retries} retries:\n{result.stdout}")

 return "" # unreachable

The Prompt Pattern That Makes Code More Validatable

The way you prompt the agent significantly affects how validatable the output is. This structure reduces hallucination in generated code by giving the model explicit constraints:

Write a Python script that [TASK DESCRIPTION].

Requirements:
- Python 3.11+
- Use only these packages: anthropic>=0.25, requests, python-dateutil
- Do NOT use: eval(), exec(), subprocess with shell=True, pickle
- API credentials must come from os.environ — never hardcoded
- Use the Anthropic SDK v0.25+ syntax: client.messages.create(model=..., max_tokens=..., messages=[...])
- Include a main() function and if __name__ == "__main__": guard
- Include a brief comment at the top listing all external dependencies

Output only the Python code. No markdown fences. No explanation.

The key additions: explicit package versions, explicit forbidden patterns, explicit API syntax example. Each of these directly addresses one of the 5 failure modes.

When a Check Fails: Retry vs. Escalate

Retry with error context (checks 1–3)

Syntax errors, missing imports, and deprecated API calls are almost always fixable by the agent in a follow-up attempt. Inject the exact error message back into the prompt:

Your previous code failed validation:
❌ [IMPORTS] 2 import(s) not found: anthropic_tools, openai_streaming
❌ [API_METHODS] client.complete() — use client.messages.create()

Rewrite the script fixing these issues. Use only packages in the standard library plus: anthropic, requests.

Escalate to human (checks 4–5)

Security issues and dry run failures are escalation triggers. An agent retrying on a security flag might just obfuscate the same pattern. A dry run failure often indicates the code depends on environment state the agent can't see. Both need human review before proceeding.

Real Example: A Validation Failure and Fix

← Back to Library