Who tests the tests?
An AI agent writes a smart contract. Another agent writes its tests. Every test passes.
That sounds reassuring. But what if both agents made the same mistake about who should be allowed to withdraw?
A passing test tells us that an assertion held under the conditions tested. It does not tell us whether those conditions cover the contract's intended behavior, or whether the assertion was worth checking.
For teams using AI to build financial software, this is more than a testing problem. It is a training opportunity: turn expert corrections into examples and checks that help agents learn what good work actually looks like.
What Mercor shows about expert feedback
Mercor describes a model of AI development in which experts help create tasks, realistic working environments and automated graders. These graders provide signals about whether an agent completed a task correctly. Its discussion of reinforcement learning connects those signals to training, rather than treating human review only as a final approval step. Welcome to the Era of Evals.
Mercor also describes evaluation systems that combine domain expertise with verifiers to measure agent behavior and outputs. Agent Eval Systems.
Our proposal is to apply that general approach to smart contract testing and preparation for security review. This is a dOrg design proposal, not a Mercor product announcement or a partnership with Mercor.
Smart contracts make the question concrete
OpenAI and Paradigm's EVMbench evaluates agents on finding, fixing and exploiting smart contract vulnerabilities in controlled environments. Its patch tasks check both whether a vulnerability is removed and whether intended functionality survives. The authors also acknowledge that evaluation remains imperfect, including difficulty judging additional findings as valid vulnerabilities or false positives. Introducing EVMbench.
That distinction matters. An agent can make a security test pass by preventing everyone from withdrawing. The exploit stops, but so does the product.
We need to evaluate the defect, the proposed fix and the evaluator itself.
A withdrawal example
Consider a fictional vault with three requirements:
- Only the owner can withdraw the owner's funds.
- An authorized withdrawal of an available balance succeeds.
- An unauthorized attempt leaves balances unchanged.
A weak test checks that the owner can withdraw. It never checks another caller. A contract missing authorization could pass.
A better test checks both callers and verifies the resulting balances. A better evaluation also runs those tests against a deliberately defective version and a version that rejects all withdrawals. The first should be caught for missing authorization; the second should be caught for breaking legitimate use.
This is an illustrative example, not a reported result from an AI model or a deployed vault. It gives us a precise experiment to build without using a client's private code or moving real funds.
Try the verifier in Test the Tests Lab
The accompanying interactive prototype computes this example in your browser. Start with Missing authorization and the weak suite: it passes despite allowing a stranger to withdraw. Switch to the specification-based suite: the unauthorized withdrawal fails the check. Then select All withdrawals disabled to see why preventing an exploit is not enough if legitimate withdrawals stop working.
The matrix compares all three versions. You can download an illustrative task record containing the specification, candidate, expected and observed outcomes, provenance and limitations. This is deterministic JavaScript simulation, not Solidity compilation, EVM execution or an AI training run. Independent expert review is still pending. These public demonstration cases would be excluded from a future held-out evaluation.
Turn a correction into a training asset
The proposed workflow starts with a specification written and reviewed by engineers. The model should not define its own task, generate its own answers and be the only judge of success.
For each case, we would assemble the contract, intended behavior, relevant assumptions, a reviewed correction and executable checks. We would also record why an attractive but incorrect answer fails. A false positive is useful too: the agent should learn when a suspicious pattern does not violate the stated requirements.
These materials could support three different kinds of improvement:
Learning from reviewed examples. A training team can use accepted fixes, tests and concise explanations as supervised examples, provided it has permission to use the material. Rejected answers can supply comparison data when the reason for rejection is clear.
Learning from executable outcomes. Where correctness can be checked reliably, a model developer can use an isolated environment and verified outcomes as reinforcement learning signals. Passing a compilation check alone should not earn the same credit as removing a defect while preserving valid behavior.
Improving the workflow without retraining. The same checks can guide a coding agent's iterations, inform tool use and catch regressions. This can improve the system's output without changing the model's parameters. It is not evidence that the model itself has been trained.
Giving an agent feedback in a chat does not automatically retrain it. Training requires a supported training process, appropriate data rights and a separate measurement of what changed.
Do not teach the model to pass a bad exam
If a grader rewards deleting the withdrawal function, we can reinforce behavior that makes the contract less useful. If a test encodes the same wrong assumption as the generated code, we can reinforce the original mistake.
We would therefore test the tests: introduce controlled defects, verify that relevant checks catch them, and confirm that valid implementations still pass. Some changes are behaviorally equivalent; a test should not be penalized for failing to distinguish an equivalent variant.
Engineers would review disagreements and cases where executable checks cannot settle the security question. Ambiguous specifications need clarification, not an invented label of “correct.”
We would also keep a separate evaluation set, including different code families and unseen variations. Training on a case and then celebrating performance on that same case measures familiarity, not useful generalization. Public examples can explain the method; private evaluation cases can help test whether it transfers.
A small pilot worth measuring
Our proposed pilot would start with 3–5 original or appropriately licensed Solidity cases, expanding to 10–15 only after the verifiers and review process prove usable. It would cover permissions, upgrades and payment accounting. Each case would include reviewed behavior, checks for defects and checks for legitimate use. All execution would stay in an isolated local environment.
We would measure:
- Known defects detected and valid cases incorrectly flagged.
- Fixes that remove the defect without breaking intended functionality.
- Incorrect solutions accepted by the evaluator.
- Human review time per accepted case.
To claim a training improvement, the pilot would need a before-and-after comparison on a held-out set under comparable model and tool settings. Improvements from extra retries, better prompts or additional tools should be reported separately from improvements caused by training.
A small pilot can establish whether the method is useful. It cannot establish that an agent is ready to audit every contract or that production code is secure.
Evidence a sponsor can inspect
We would deliver versioned task bundles, executable verifier instructions, mutation results, review decisions and a reproducible evaluation report. Each case would record its source, license, permitted training use and reviewer status. Client code would require separate, explicit permission; public visibility alone does not establish training rights.
The evaluation plan would freeze the model version, tool access, attempt budget and relevant settings, and compare against an expert baseline. We would report task-level counts and uncertainty, including failed or inconclusive experiments, rather than treating a small percentage change as proof of improvement. Different code families would be reserved for evaluation, with controls against leaking solutions into training.
An independent Solidity reviewer should be able to reject a task or challenge a verifier. Funding would pay for that review as well as engineering. The deliverables, reviewer allocation, costs and stop-or-expand criteria would be agreed before work starts. This follows the emphasis on expertise, reproducibility, expert baselines and small initial task sets in Anthropic's published evaluation principles; it does not imply acceptance by that program.
Where an engineering partner fits
An AI team might have the training infrastructure but need specialists to create Solidity tasks and reliable checks. A security-tool company might need independent evaluation of findings and fixes. A protocol team might need a better testing workflow before its formal audit.
Those are different engagements. The first sells expert training materials; the second sells evaluation; the third sells engineering and preparation for review. None should be presented as a guarantee of security or a replacement for an independent audit.
At dOrg, we are looking for a funder or technical sponsor to support a scoped pilot that builds and reviews these cases. Funding would support engineering, independent review and evaluation infrastructure, with a bounded training experiment if an appropriate model developer and training setup are available. The valuable deliverable is not a pile of AI-generated tests. It is a set of defensible examples and checks that a team can use, inspect and reproduce.
Our starting point is hands-on Web3 engineering, not a claim that we have already trained a security model. You can inspect dOrg's public engineering work before discussing the pilot. A sponsor would receive a named delivery and independent-review plan before agreeing to fund the work.
What we are seeking funding to build
The first milestone would be a small set of original Solidity tasks, reviewed solutions and verifiers that catch both security defects and broken functionality. The next would be a baseline evaluation of an existing agent. A training phase would follow only after agreeing on model access, data rights, a supported training method and the cost of the experiment.
The research question is specific: can expert-reviewed training tasks improve performance on unseen smart contract testing and repair cases, without simply increasing retries or teaching the agent to pass familiar tests?
We would agree on deliverables, budget and success criteria with the sponsor before beginning. A public-goods grant could support openly licensed materials and reproducible research. A commercial sponsor could commission a private evaluation or training-data engagement under separately agreed rights. Credits can help pay for model use, but engineering and review need funding too.
We are not raising capital for an already trained model, and we are not claiming a measured improvement yet. We are seeking support to build and test the approach.
Bring one failure mode
Do you fund AI evaluation research, Ethereum security tooling or expert training data? We would like to discuss supporting this pilot. If you develop an AI coding or security tool for Solidity, bring one recurring problem: missed authorization, invalid findings, brittle tests or fixes that break valid behavior.
We would start by agreeing on what a correct outcome means, then scope a funded pilot to evaluate it. If training is the next step, we can prepare the expert-reviewed materials for a suitable training pipeline and measure the results separately.
Discuss funding this pilot with Fernando at dOrg. Tell us whether you fund public research or commission private training and evaluation work, and which failure mode matters to your team. We can then prepare a milestone proposal with explicit deliverables, costs, rights and review requirements.
Better AI security starts with better definitions of success. Someone still has to make sure the tests deserve to be trusted.
Cover: fictional editorial illustration of Vitalik Buterin, Alan Turing and the dOrg mascot. It does not imply endorsement, participation or funding.