The story centers on a Claude Code–driven, open-source system for fully automated prompt optimization (FAPO) that designs and optimizes multi-step LLM pipelines, with guardrails to prevent overfitting and leakage and a three-stage escalation for failures.

Use cases include multi-hop question answering, instruction following, and classification, with iterative optimization yielding measurable gains in validation and test accuracy.

The optimization loop runs through six stages each cycle—Evaluate, Attribute, Propose, Review, Compare, Iterate—while escalating at three levels: prompt, parameter, and chain structure.

FAPO stands for Fully Automated Prompt Optimization, a Claude Code–driven system that autonomously optimizes LLM pipelines from baseline prompts to target accuracy, released as open source under Apache 2.0.

Benchmarks, setup instructions, and an interactive explainer accompany the article, with links to the GitHub repository and a technical blog for deeper detail.

Getting started involves Claude Code scaffolding to generate tenant files from task descriptions and a JSONL dataset, then closed-loop evaluation and optimization via the hephaestus engine, culminating in a final one-shot evaluation on held-out data.

Cisco’s benchmarks show FAPO outperforming GEPA in most model-benchmark comparisons, with substantial mean gains and larger improvements when structural changes are required.

Step-level failure attribution categorizes errors into retrieval, cascade, format, and reasoning, guiding targeted prompt, parameter, or chain-structure adjustments.

Guardrails are built to prevent overfitting and leakage, including training-split-only evaluation, immutable variant files, and independent reviewer validation of every proposal before execution.

FAPO supports multiple providers (OpenAI, Baseten, SageMaker) and uses LangGraph-based chains to process test cases, needing only a dataset and an initial prompt scaffold from Claude.