A plaster cast resting in its mould, both sharing the same chipped corner, next to a reference block with its corner intact.

5of 6

Agentic engineering: define success outside the code

Don’t let an agent’s own tests be the only judge of its code. Define what must work, what the software must reject, and who can approve a release.

Build & Validate
Evidence-led buildingAI & Automation

Adapted by the EWOR Team from Paul Müller’s article (opens in a new tab)2 min read

Imagine an agent writing a feature, creating its own tests and reporting that everything passes. It may have made the same mistake twice.

That’s the risk EWOR Partner and Adjust co-founder Paul Müller highlights in Stop Watching Your Agents Code. If an agent misunderstands a requirement, that misunderstanding can carry through to both the implementation and the tests designed to verify it. A green test run may look reassuring, but it tells you less than you think.

The answer isn’t to abandon unit tests. It’s to define success independently of the implementation and have someone other than the agent check whether the result meets that standard.

Start with one bounded change

Describe what a user or connected system should observe. Include expected failures, actions the software must reject and conditions under which the change must not ship.

Set the working boundaries too:

  • Which interfaces may change,
  • Which conventions matter, and
  • Which parts of the system must stay untouched.

This reduces ambiguity without prescribing every line of code.

Test what the software does and what it should refuse to do

Keep your usual checks, but don't rely on them alone. Add acceptance tests based on observable behaviour: given a particular input or interaction, what output, state change or rejection should follow?

For a hypothetical permissions feature, a pair of cases could be:

  • A user with permission can retrieve the permitted record.
  • A user without permission is denied access and receives none of that record’s protected data.

Those cases illustrate the distinction; they are not a complete security test plan. Add the adverse cases that matter for the actual feature. A performance-sensitive path also needs a relevant threshold, not just a functional success check.

Have someone independent of the implementing agent review the requirements and tests themselves. A test can pass perfectly while checking the wrong thing. Passing tests tell you something about the scenarios you've covered; they don't prove the software is complete, correct in every situation or secure.

Let the agent iterate inside limits

You don't need someone watching every keystroke for an agent to be useful. But you do need boundaries around what it can access and change.

Run the work in an isolated environment, with only the credentials and tool permissions it needs. Production data, privileged tools, infrastructure changes and external side effects should require explicit approval. Asking an agent to implement a feature doesn't automatically authorise it to make those changes.

Before release, review the code changes, dependencies and security implications. Keep deployment approval with a person, and make sure monitoring and rollback are in place. The agent can take on more of the implementation, but accountability for the result still belongs to the team.

Source: EWOR article by Paul Müller (opens in a new tab)

No comments yet.

Be the first to comment on this signal.

Get the next signal before everyone else.

Frameworks and contrarian theses for outlier founders, straight to your inbox. No fluff, no recycled advice, weekly.

The EWOR Signals newsletter

Continue learning

Related modules