Agents Can Only Fix What They Can Detect

I've been spending a lot of time building Continuum Atlas, a deterministic renderer for software architecture diagrams.

One of the challenges is getting diagrams to look good without compromising their accuracy. That's harder than it sounds.

Atlas evaluates different layouts using measurable properties: how many lines cross, how many bends they contain, how elements align, how they're spaced, and how well the diagram fits within its available space.

The idea is straightforward. Generate several candidates, measure them, and select the best one.

At least, that's what I thought.

I had a diagram where none of the five layouts generated using ELK's connector routing passed validation. After replacing that routing with Atlas's own obstacle-avoiding A* algorithm, all five passed.

That was a significant improvement.

But when I looked at the results, one of the newly valid layouts was actually worse. It had less measurable clutter, yet it was noticeably harder to read.

The measurements weren't wrong. The algorithm had genuinely improved the results according to the properties we were measuring.

The problem was that we weren't measuring everything that mattered.

I've been thinking about that experience because it highlights something important about agentic AI.

An agent can only correct a problem if something in its feedback loop can recognize that the problem exists.

What makes an agent an agent?

There's a lot of discussion about AI agents right now, and the terminology can make something relatively straightforward sound unnecessarily complicated.

At its core, an agent works through a feedback loop.

It observes what's happening, reasons about what to do next, takes an action, evaluates the outcome, and uses what it learns to decide what to do next.

Figure 1. The agentic loop. The result of each action becomes evidence that informs the next decision.

Consider asking a language model to fix a software defect.

Without tools, the model can examine the code you provide, suggest a fix, and explain why it believes the change will work.

That can be useful, but there's an important limitation. The model hasn't actually established that the defect is fixed.

Now give the model access to the repository, a compiler, and a test runner.

It can inspect the implementation, make a change, compile the code, run the tests, examine failures, and adjust its approach.

If the tests fail, it has evidence that something still needs attention.

Instead of simply producing an answer, the system can work toward an outcome.

That's what makes the agentic loop interesting to me.

But there's an assumption hidden in this process.

We're assuming that the feedback is capable of detecting the things we care about.

And that isn't always true.

When the feedback isn't enough

I've started thinking about agentic systems in three different situations.

The first is when there's no feedback.

The model generates an answer, and someone has to decide whether that answer is correct. We've been working this way with language models for years.

The second is when the system has meaningful feedback.

A compiler rejects the code. A test fails. An architecture constraint is violated. The agent sees the result and tries to correct it.

This is where agentic systems become genuinely useful. They can work through problems, learn from unsuccessful attempts, and make progress without requiring someone to direct every step.

The third situation is more interesting.

What happens when all the checks you've performed look good, but you've missed something important?

I've encountered variations of this while building Continuum.

In one case, we had a generated architecture model and a compile-proof attestation that were consistent with each other. The verification was legitimate. It established that the model compiled successfully.

But a subsequent operation failed because of an issue in the component responsible for preserving the model's representation during emission.

The compiler hadn't made a mistake. The attestation hadn't made a false claim.

We had simply established one property of the system and assumed it told us something about another.

I've seen a related problem with OWL, the AI code-review system I discussed in From Prompt to Platform.

During three evaluation runs, the reviewer returned ApprovedWithComments each time. Yet the aggregation process encountered parsing failures in two of those runs, and the reviewer wasn't consistently identifying the same known, actionable defects.

That raises an important question.

If an AI reviewer completes its review and produces a reasonable-looking result, how do we know the review was actually effective?

We need something beyond the reviewer's own assessment.

And even then, we need to understand what that additional verification actually establishes.

A passing test tells us the tests passed. It doesn't tell us the implementation satisfies requirements the tests never covered.

An architecture checker can establish that a system complies with the rules we've defined. It can't tell us whether we've defined the right rules.

A successful code review doesn't necessarily mean every important defect was identified.

These distinctions may seem obvious when you examine them individually.

They're much easier to overlook when an agent has been working autonomously for twenty minutes and presents you with a list of successful checks.

It's tempting to interpret that list as evidence that the entire task was completed correctly.

Sometimes it is.

Sometimes it's evidence that the agent successfully completed the parts of the task we knew how to check.

Those are two different things.

Building the things that can say no

One of the things I've learned from building Continuum is that making an AI system reliable requires a surprising amount of engineering outside the language model.

The model is only one part of the system.

We've invested considerable effort into making architecture specifications machine-readable, validating constraints, checking invariants, and defining acceptance criteria.

The idea is to give an agent something authoritative to work against.

If an agent modifies an implementation, we want to be able to establish whether the change remains consistent with the intended architecture.

We don't want the model to be the only thing deciding whether its own work is correct.

With Continuum Atlas, we're applying a similar principle to diagrams.

Atlas generates multiple layout candidates and evaluates them using deterministic measurements. We can compare candidates, identify trade-offs, and reject layouts that violate specific constraints.

We also distinguish between three things: the measurements themselves, quality scores derived from those measurements, and human judgment.

That distinction matters.

A diagram might have fewer crossings and better alignment while still being more difficult to understand.

We shouldn't pretend that combining several measurements into a single score somehow captures everything about visual quality.

Instead, we can use the measurements to narrow the choices and make the trade-offs visible.

The final judgment may still require a person.

And I think that's perfectly reasonable.

The goal isn't to automate every decision. It's to automate the decisions we can evaluate reliably while being explicit about the ones we can't.

How much autonomy can we really give an agent?

This has changed how I think about autonomy.

A lot of the discussion around agentic AI focuses on model capability.

Can the model reason more effectively? Can it handle larger tasks? Can it use more tools? Can it work independently for longer?

Those improvements matter.

A more capable model can explore better solutions, recover from mistakes, and accomplish things that less capable models struggle with.

But greater capability doesn't automatically mean we should grant greater autonomy.

Consider an agent that's allowed to modify code, run tests, and prepare a pull request.

We can give it considerable freedom because there are mechanisms to inspect its work before the change reaches production.

Now consider allowing the same agent to deploy directly to production.

The model hasn't necessarily changed. What changed is the consequence of an incorrect decision.

Before granting that additional authority, I'd want to understand what checks are in place, which risks they cover, and what happens when something unexpected occurs.

The question becomes less about what the model is capable of doing and more about what the system can safely allow it to do.

To me, that's an architectural decision.

And it's one of the reasons I don't think agentic AI can be reduced to better prompts or more powerful models.

You need to design the environment around the model: the tools it can access, the constraints it must respect, the evidence it needs to produce, and the conditions under which its work can be accepted.

The limits of what we can measure

This brings me back to the diagram that started this whole line of thinking.

We improved the routing algorithm. We went from zero valid layouts to five.

That was real progress.

But one of the resulting diagrams was still worse to look at.

We could have responded by adding more metrics, tweaking weights, and trying to make the quality score reflect our preferences more closely.

Some of that would probably help.

But there's a danger in assuming that every judgment can eventually be reduced to a number.

Software engineering is full of decisions that are difficult to evaluate mechanically.

Is this the right abstraction?

Is the architecture unnecessarily complicated?

Will another developer find this API intuitive?

Is this actually the feature we should be building?

We can collect evidence to inform those decisions. We can define constraints and evaluate alternatives.

But we shouldn't confuse having measurements with having a complete understanding of the outcome.

That's something I'm trying to be more deliberate about in Continuum.

I want agents to do as much as they reasonably can on their own. I also want to know where the verification ends and human judgment begins.

The better we become at defining and checking the outcomes we care about, the more autonomy we can safely give these systems.

And where we can't reliably check an outcome, we need to acknowledge that limitation rather than hide it behind a successful execution report.

What did we actually verify?

I still think the agentic loop is one of the most important ideas in AI-assisted engineering.

Giving a model the ability to act, observe results, and correct its mistakes changes what we can accomplish with it.

But building these systems has made me appreciate that feedback is only useful to the extent that it tells us something meaningful about the outcome.

You can have an intelligent model, a well-designed loop, and a complete record of every action taken.

And you can still end up with the wrong result.

That's why I've become increasingly interested in the engineering that surrounds the model, particularly the mechanisms that independently evaluate its work.

When an agent tells me it has finished a task, I'm less interested in how confidently it explains what it did.

I want to know what actually happened, what was checked, and what remains uncertain.

Because a green check tells you something passed. The important question is what, exactly, it proves.