AI smart contract audits: what we learned from 50 runs

AI Smart Contract Audits: What the Benchmark Runs Taught Us

The difficult question is whether that description survives contact with an unfamiliar contract, a multi-step exploit path, and the operational pressure of a real security review.

Recent evaluations draw a sharp line between those tasks. Frontier models can identify some known vulnerability patterns, especially when the benchmark resembles material that may have appeared in their training data. Performance becomes less dependable when the model has to construct an exploit against a recently observed contract, reason across several contracts, or distinguish a genuine invariant break from a superficially similar code pattern. Add complex DeFi logic and the problem changes again: the issue is no longer only missed vulnerabilities, but the volume of findings that an auditor must reject.

The available record does not support a single accuracy number for AI-assisted security work. It describes several different systems operating under different conditions: EVMBench and its contamination concerns, CyberChainBench’s real-world incident corpus, the out-of-distribution ReEVMBench evaluation, and the SCONE-bench simulation. Read together, they suggest a narrower but more useful conclusion. AI can contribute to a security pipeline, but the contribution depends on how much deterministic analysis and human judgment surround the model.

The Benchmark Reality: EVMBench and the Contamination Problem

On February 18, 2026, OpenAI, Paradigm, and OtterSec released EVMBench, a benchmark comprising 117 high-severity smart contract vulnerabilities collected from 40 public audit competitions. The benchmark was designed as a ground-truth evaluation for AI agents working with Solidity source and EVM-related security tasks. Its vulnerability set came from contests conducted between 2024 and 2025.

That time window is central to interpreting the results. The benchmark material was public, and it preceded the training cutoffs of the frontier models under evaluation. This creates a credible contamination risk: a model may have encountered the source material, a report describing the vulnerability, or a close paraphrase of the exploit before the benchmark was assembled.

That risk does not prove that any particular model memorized the answers. It does mean that the benchmark cannot be treated as a clean measurement of generalization without additional controls. A high score may reflect some combination of reasoning, pattern recognition, exposure to related examples, and direct or indirect familiarity with the underlying public material. The benchmark is still useful, but its headline figures need to be read as results on a potentially contaminated distribution rather than as a definitive measure of performance on novel contracts.

When frontier models were evaluated without external tool augmentation, reported detection rates ranged from 23.3% for GPT-5 to 45.6% for Claude Opus 4.6. Intermediate models fell between those figures. The benchmark counted a vulnerability as detected when the model’s classification met the required severity standard in the grading rubric.

Specialized multi-agent security frameworks exceeded 87% detection under the same benchmark conditions. That is a substantial result, but the comparison needs to be stated accurately. Against the strongest unassisted baseline at 45.6%, the lift is greater than 41.4 percentage points. Against the 23.3% lower bound, it is greater than 63.7 points. It is not an eight-to-fifteen-point improvement.

The reason for that difference is architectural. A framework that combines an LLM with static analyzers, symbolic execution, retrieval, test generation, or dedicated triage agents is not measuring the language model in isolation. Its score reflects the complete system. That distinction matters when a team is deciding whether to buy an AI smart contract security scanner or build an internal workflow around a general-purpose model.

The contamination concern was raised publicly by OpenZeppelin shortly after EVMBench’s release. Their March 2, 2026 analysis noted that the benchmark’s curated vulnerabilities came from public contest audits that predated the training cutoffs of the major models being tested. The observation does not establish memorization as the cause of the reported scores. It establishes that memorization and benchmark leakage are plausible confounders that should be controlled for in future evaluations.

A clean evaluation would need more than a new prompt. It would need contracts and incidents that were not available to the models before their cutoff, careful provenance checks, and a separation between vulnerability detection and exploit reproduction. It would also need to report failures in enough detail to show whether the model missed the bug, identified the wrong root cause, or found the right issue but could not express it in a form the grader accepted.

A contaminated benchmark can still reveal useful engineering differences, but it cannot by itself prove that a model will generalize to an unseen contract.

For teams deploying automated smart contract audit tools, this changes how benchmark results should be used. EVMBench may help compare two systems under a shared test harness. It should not be used as a production guarantee, and especially not as evidence that a model will discover novel exploit paths in code released after its training window.

The distinction is familiar from other areas of machine learning, but it is unusually consequential in security. A model that has seen a vulnerability pattern can produce an impressive explanation while still failing to reason about the next variation of that pattern. In an audit, the variation is often where the money is: a different storage layout, an unexpected callback, a non-standard proxy, a governance delay, or an oracle update that changes the economic meaning of an otherwise ordinary function.

Frontier Model Performance: From Detection to Exploit Synthesis

“Detection” and “exploitation” are not interchangeable benchmark tasks.

Detection asks the system to identify a suspicious location or classify a vulnerability. Depending on the evaluation, it may need to name the issue, assign a severity, and explain the relevant code path. Exploit synthesis asks for more. The agent must construct a sequence of actions that reaches an invalid state or extracts value while respecting the contract’s permissions, balances, call ordering, and external dependencies.

That additional requirement exposes weaknesses that a label can conceal. A model may recognize the words associated with reentrancy without finding a viable reentrant path. It may flag an access-control concern without identifying a caller that can actually reach the privileged branch. It may describe an oracle manipulation scenario without showing that the relevant price can be moved far enough, long enough, and at an acceptable cost to make the attack work.

CyberChainBench, published to arXiv in June 2026, evaluated LLM security agents across 541 real-world exploit incidents from nine EVM chains. The top-scoring agent recorded 37.5% on detection, 43.7% on exploitation, and 23.4% on patch synthesis.

The ordering matters. Exploitation, at 43.7%, was higher than detection, at 37.5%, in the reported headline figures. That does not mean exploit synthesis is generally easier than detection. The metrics may involve different task definitions, prompts, success criteria, or subsets of incidents. It does mean that the figures cannot be presented as evidence that detection consistently outperforms exploitation. They should be treated as separate measurements rather than arranged into a universal hierarchy.

Patch synthesis was lower than both of those figures. This is the operationally important part. Finding a suspicious line is not the same as producing a safe remediation. A patch has to preserve intended behavior, maintain storage compatibility where relevant, respect upgrade and governance constraints, and avoid introducing a second vulnerability. In DeFi, it may also need to preserve economic assumptions that are not explicit in the Solidity code.

CyberChainBench’s corpus contains real-world incidents, but the supplied record does not establish how those incidents were distributed relative to the training data of the evaluated models. Some incidents may have been public before the relevant cutoffs; others may have been less represented or described in different terms. It is therefore reasonable to discuss possible distribution overlap as a risk, but not to attribute the 43.7% exploitation result to a known training-window distribution.

A model can perform well on a mixed corpus for several reasons:

• It may recognize recurring exploit primitives from training.
• It may infer the attack from the code and the prompt without having seen the specific incident.
• It may benefit from scaffolding that decomposes the task into smaller steps.
• It may exploit weaknesses in the benchmark’s task format.
• It may combine genuine reasoning with memorized security language or incident descriptions.

The benchmark score alone does not separate those explanations.

SCONE-bench provides a different kind of evidence. In a simulation experiment conducted by Anthropic researchers in December 2025, agents examined 2,849 recently deployed smart contracts with no known bug history at the time of sampling. The agents found two novel zero-day vulnerabilities. The reported exploit revenue was $3,694 against an API execution cost of $3,476.

Two findings are possible here, and both need to remain in view. First, the experiment shows that an AI agent can produce novel exploit discoveries against a corpus selected to avoid known bug history. That is evidence against the strongest version of the claim that model-based exploit synthesis is limited entirely to memorized incidents. Second, two discoveries are not enough to establish a stable success rate or a reliable business model. The revenue and API-cost comparison is illustrative, not a statistically strong estimate of future returns. It also excludes the engineering work required to validate a finding, reproduce it safely, assess disclosure obligations, and implement a fix.

SCONE-bench was a simulation experiment, not a live-deployment result. That distinction matters. A simulated agent may have different permissions, latency, observability, and transaction constraints from a production monitoring system. Still, the result is relevant because it demonstrates a workflow that is often missing from contaminated benchmark discussions: search over recent contracts, generation of candidate exploit paths, and validation against an unseen target set.

The practical interpretation is cautious. Novel discovery is possible, but rare findings do not turn an agent into an autonomous auditor. They show that the system can sometimes leave the familiar pattern space. They do not show that it can do so with predictable coverage, bounded false positives, or production-grade safety.

The 97% False-Positive Barrier in Complex DeFi Logic

Detection rates receive the most attention because they are easy to place in a table. For an audit team, false positives may be more important.

A scanner that reports a plausible issue in every function can look active while making the review slower. The auditor must determine whether the reported state is reachable, whether the attacker controls the relevant input, whether an oracle or external call changes the path, whether the protocol’s invariant is actually broken, and whether the finding is already mitigated elsewhere. Every incorrect alert consumes some of the same scarce attention needed to investigate a real one.

Empirical assessments of unguided LLM tools on complex DeFi codebases have reported false-positive rates above 97%. The difficult cases include cross-file interactions, multi-contract proxy systems, oracle-dependent invariants, liquidation logic, governance paths, and business rules that cannot be inferred from a single function. The point is not that every AI scanner produces exactly the same rate in every setting. The point is that the false-positive problem becomes severe when the evaluation moves beyond isolated, recognizable code patterns.

DeFi contracts are difficult for a specific reason: their security properties are often relational. A function may be safe only because another contract updates a value first. A withdrawal may be valid only after a debt calculation, a price freshness check, and a governance delay. A liquidation may appear to transfer assets at an unusual discount because that discount is part of the protocol’s intended incentive design. A model that reviews one file in isolation can identify a suspicious pattern while missing the system-level condition that makes it safe—or the condition that makes it exploitable.

The model’s output is also sensitive to the way the problem is framed. Ask for vulnerabilities, and it may optimize for recall-like behavior by producing a broad list. Ask for confirmed exploitable issues, and it may become more conservative while missing subtle paths. Ask for a severity ranking, and it may assign confidence based on linguistic familiarity rather than a verified state transition. These are not cosmetic prompt differences. They change the operating point of the system.

A useful AI smart contract audit therefore needs a gate between generation and acceptance. That gate can include:

• Static analysis to identify reachable patterns and suspicious data flows.
• Symbolic execution to test whether proposed paths satisfy their conditions.
• Fuzzing to explore concrete state transitions and edge cases.
• Call-graph and dependency analysis across contracts and libraries.
• Transaction simulation for candidate exploit sequences.
• Human review of protocol-specific invariants and economic assumptions.
• A finding schema that separates evidence, hypothesis, severity, and remediation.

Without those controls, an LLM may turn uncertainty into a polished narrative. The explanation can be technically fluent while the underlying claim remains unverified.

In complex DeFi, the cost of a false positive is not only a wasted click. It is the opportunity cost of sending an auditor down the wrong execution path.

Specialized multi-agent frameworks address part of this problem by assigning different tasks to different components. One agent may propose vulnerabilities, another may challenge the claim, and a tool layer may attempt to reproduce the path. A separate triage stage can compare the finding against known false-positive patterns. This is a more defensible design than asking one model to produce a complete audit report from source code alone.

But the reported EVMBench performance of more than 87% detection should not be read as proof that these frameworks have solved false positives in real-world DeFi. The result demonstrates a strong benchmark advantage under the stated conditions. It does not provide, by itself, a false-positive rate for every production setting, nor does it establish that the language model is responsible for the full improvement. The deterministic tools and orchestration layer may be doing much of the work.

That is not a criticism of the framework. It is the central engineering lesson. If a system works because it combines probabilistic reasoning with verifiable analysis, then the combination—not the model’s standalone score—is the product that should be evaluated and deployed.

Why Real-World Exploit Success Remains at Zero

BlockSec’s ReEVMBench was designed to examine the generalization problem that EVMBench could not resolve. Published in March 2026, it used 110 agent-incident tests drawn from 22 real-world exploit incidents that occurred after February 2026. The end-to-end exploit success rate was 0%.

This is the cleanest negative result in the record, but it also needs to be scoped correctly. ReEVMBench shows that the evaluated agents did not successfully complete the end-to-end exploit task on that post-cutoff set. It does not prove that all AI agents are incapable of novel exploitation, and it does not establish that every failure arose from the same limitation. Agents can fail because they miss the vulnerability, misunderstand the contract state, construct an invalid transaction sequence, lack the required tooling, or stop before validating an otherwise promising path.

The result is also not directly comparable to every detection score. Detection, exploitation, and patch synthesis require different outputs. A system can flag the right function but fail to produce an executable exploit. Conversely, a benchmark may provide enough context for an agent to search for an exploit even when its vulnerability explanation would not satisfy a detection grader. The metrics should be kept separate.

The supplied evidence does support a more limited conclusion: performance on known or potentially familiar distributions should not be projected onto post-cutoff exploit discovery. The gap between CyberChainBench’s 43.7% exploitation score and ReEVMBench’s 0% end-to-end success is a warning about transfer. It is not proof that the earlier score was memorization, because the training-distribution relationship has not been established. It is evidence that benchmark context and data freshness can materially change the result.

The same caution applies to claims about the model’s corpus being a closed set. SCONE-bench found two novel zero-day vulnerabilities in a simulation involving 2,849 recently deployed contracts. That result contradicts any absolute claim that an agent can only reproduce exploits already represented in its training material. The more defensible statement is that language-model-based exploit synthesis may be strongly affected by prior exposure, while novel discovery remains possible but difficult to quantify from a very small number of successful cases.

The table should be read as a map of task conditions, not as a league table. EVMBench gives a detection comparison but carries a contamination concern. CyberChainBench measures detection, exploitation, and patch synthesis on a broad incident corpus, but the relationship between that corpus and model training data is not established. ReEVMBench isolates a post-cutoff exploit setting and reports no successful end-to-end exploits. SCONE-bench demonstrates that novel discovery can occur in simulation, but its two findings do not provide a general success rate.

Those distinctions matter for procurement and architecture. A vendor presenting one number as the accuracy of its AI security scanner is compressing several different questions into a metric that may not answer the one an engineering team actually has. The team may need to know whether the system can find an access-control flaw in a new contract, validate an oracle manipulation path, generate a safe patch, or monitor a deployed protocol for a concrete exploit sequence. Each is a different capability.

The most dangerous extrapolation is from benchmark recognition to autonomous defense. Recognizing a previously documented issue is useful. It is not the same as discovering a new issue in a protocol whose assumptions, dependencies, and economic incentives differ from the examples in the benchmark.

The Future of Hybrid Security: Multi-Agent Frameworks vs. Human Triage

The empirical record does not justify abandoning AI-assisted auditing. It does justify abandoning the idea that a general-purpose model can serve as a complete security pipeline.

A useful architecture separates the jobs instead of asking one model to perform all of them. The LLM can help enumerate hypotheses, summarize code paths, translate findings into a common schema, generate candidate tests, and suggest places where an auditor should look next. Deterministic tools can check reachability, storage behavior, arithmetic constraints, and transaction sequences. Human auditors can decide whether the behavior violates the protocol’s intended invariants and whether a proposed fix preserves them.

In practice, the boundary between those layers should be explicit. A model-generated finding should not become a confirmed vulnerability merely because the explanation sounds confident. It should carry evidence: the relevant path, assumptions about caller permissions, required state, external dependencies, and the result of any reproduction attempt. If the system cannot validate those elements, the finding should remain a hypothesis.

Multi-agent frameworks are promising because they can distribute this work. A planning agent can decompose the contract. A code-analysis agent can identify candidate weaknesses. A symbolic or fuzzing tool can test them. A critic agent can search for counterexamples. A final triage layer can decide whether the evidence is strong enough for human escalation. The architecture reduces the chance that a single fluent but incorrect answer passes directly into an audit report.

It does not remove the need for human triage. Protocol-specific intent is often not encoded in the source code. The auditor may need to understand the oracle’s trust model, the expected behavior during a market halt, the relationship between governance and emergency powers, or whether a seemingly unusual transfer is an intentional part of liquidation economics. Those judgments are not reliably recoverable from syntax and generic vulnerability patterns.

The economics are similarly conditional. SCONE-bench’s reported exploit revenue exceeded its API execution cost by only a modest margin, and the experiment involved two novel discoveries. Once validation, infrastructure, researcher time, responsible disclosure, and remediation are included, the return calculation becomes more difficult. A security team may still accept those economics because the value of preventing a loss is much larger than the revenue captured by an offensive experiment. But that is a different business case from autonomous auditing at scale.

For a development team choosing among automated smart contract audit tools, the meaningful questions are therefore architectural:

1. What does the system count as a finding?

A pattern match, a reachable path, a reproduced exploit, and a protocol-level invariant violation are not equivalent outputs.

2. Which claims are verified by tools?

The system should distinguish model reasoning from results produced by static analysis, symbolic execution, fuzzing, or transaction simulation.

3. How does it handle cross-contract state?

Support for a single Solidity file says little about proxies, callbacks, oracle updates, governance modules, and external token behavior.

4. What happens to uncertain findings?

A useful scanner makes uncertainty visible and routes ambiguous cases to review instead of presenting every hypothesis as a confirmed issue.

5. How is performance measured on fresh code?

Evaluation should include post-cutoff or otherwise controlled out-of-distribution contracts, not only public vulnerabilities that models may have encountered.

6. Can the system produce a safe remediation?

Patch suggestions require regression testing, invariant checks, and review of upgrade and storage implications. A generated diff is not a verified fix.

7. Where is the human decision point?

The workflow should define which actions require an auditor’s approval, especially when a finding could trigger an emergency response or an on-chain transaction.

These are not formalities. They determine whether the product is an AI smart contract security scanner or simply a language interface over a codebase.

The credible future of AI auditing is not an autonomous oracle of safety. It is a layered system in which the model proposes, tools test, and humans decide.

The strongest result in the current record belongs to composition. Specialized multi-agent frameworks exceeded 87% detection on EVMBench, while unassisted frontier models reached 23.3% to 45.6% on that same benchmark. The gap is larger than 41.4 percentage points relative to the best unassisted baseline, and it shows that orchestration and deterministic tooling can materially change performance. It does not show that an LLM alone has crossed the threshold for autonomous security review.

The negative results are equally useful. ReEVMBench’s 0% end-to-end exploit success on its post-cutoff tests warns against treating in-distribution performance as a proxy for novel exploit discovery. The false-positive findings in complex DeFi show why detection recall without triage quality is an incomplete product metric. CyberChainBench’s lower patch-synthesis figure shows that identifying a flaw and fixing it are separate engineering problems. SCONE-bench shows that novel discovery is possible, while also showing how little a pair of successful discoveries can establish about general reliability.

The defensible verdict is narrower than both the marketing claim and the blanket rejection. Unassisted AI smart contract audit tools have not been demonstrated as standalone security infrastructure. They can accelerate reconnaissance, explain familiar patterns, generate test ideas, and help organize a review. Their outputs still require validation, particularly when the contract spans multiple components or depends on economic and oracle assumptions.

Hybrid systems are more credible because they place the model inside a verification pipeline. Static analysis, symbolic execution, fuzzing, simulation, and human triage do not merely improve an LLM’s presentation. They constrain what the system is allowed to call a vulnerability. That is the difference between a plausible explanation and evidence that a security decision can rely on.

For now, an AI smart contract audit should be treated as a pre-filter and investigation assistant—not as a replacement for adversarial verification. The benchmark record supports that role. It does not support removing the human from the critical path.