When AI Gets Better at the Wrong Thing

I've been wondering why some AI development efforts become remarkably productive and strangely underwhelming at the same time.

You know the pattern. An agent creates a plan, writes the code, runs the tests, reviews its own work, finds more things to improve, and starts another iteration. Perhaps a second agent critiques the first. The implementation gets more sophisticated. The test count grows. The progress reports are encouraging.

Then you try the result.

Everything seems to work. Yet it isn't quite what you had in mind.

That's the rabbit hole I've been thinking about: not an AI going off task in some spectacular way, but an AI making steady, defensible improvements to something that never quite becomes the thing you wanted.

The diagram that kept getting better

I've seen a version of this while working on architecture diagrams. The challenge was to produce diagrams that were both faithful to the underlying model and pleasant to read.

A deterministic layout system could measure concrete properties: collisions, crossings, spacing, routing, and whether elements stayed within the canvas. These are useful checks. They make it possible to reject visibly broken layouts and compare alternatives without relying entirely on somebody's opinion.

In one experiment, a change to the routing algorithm improved the measured results. Layouts that previously failed validation now passed. By the system's criteria, we'd made progress.

But when I looked at one of the diagrams, it was harder to read.

The measurements hadn't lied. We'd improved what we measured. The mistake was assuming those measurements adequately represented what made a diagram good.

That experience helped change our approach. Instead of treating composition as a tightly constrained optimization problem, we began giving the composing agent more of the relevant architectural context and judging the rendered result for both meaning and visual clarity.

The lesson wasn't that constraints or automated evaluation are useless. We still needed fidelity checks. The lesson was that a better score could coexist with a worse experience.

I've started noticing the same distinction in software development.

When the reviewer shares the builder's blind spot

Imagine one AI agent building an application and another reviewing it. The reviewer checks whether the requested features are present, whether tests pass, whether the architecture is consistent, and whether the implementation follows established conventions.

These checks can be extremely valuable. For many defects, an independent reviewer with good tools is exactly what you want.

But what if the feature was specified around an incomplete idea of what the user needed?

The builder can faithfully implement that idea. The reviewer can faithfully validate it. A third agent can fix every issue the reviewer raises. All three can agree that the task is complete.

They might even be right about everything they checked.

The missing piece is the thing none of them was asked, or equipped, to observe.

There's a difference between detecting a contradiction in a specification and recognizing that the specification describes an experience nobody particularly wants. More agents don't automatically resolve that difference. Sometimes they amplify it, because each new iteration treats the original definition of success as a given.

The rabbit hole, in other words, may be perfectly rational.

The part that feels right

One of my more satisfying development experiences has been building HashRelay, a notification product I've been experimenting with.

What mattered to me wasn't simply that it could deliver a notification. I had a particular interaction in mind: an agent or integration sends a message, it reaches the device, and an approval can be handled from the notification itself rather than making the person jump through an unnecessary extra workflow.

We had to work through details that weren't obvious from the high-level feature description. At one point, approval controls weren't behaving as intended. Later, an approval action led to a web page when the experience I wanted was direct interaction from the notification. When the direct action worked, the result felt substantially closer to the product I'd imagined.

A test could tell us whether the approval API responded correctly. It couldn't, by itself, tell us that being redirected to a browser was the wrong interaction for this product.

I could.

That distinction interests me because it isn't captured by the usual definition of software quality. The implementation can be secure, fast, maintainable, and functionally correct while still missing the intended experience.

And conversely, a product that feels wonderful can still hide serious technical defects. Feeling right is not a replacement for verification. It's another kind of evidence, answering a different question.

Can AI judge what feels satisfying?

AI can make informed judgments about usability, aesthetics, and likely human preferences. It can compare alternatives, learn from feedback, and identify patterns that users themselves may struggle to articulate. In some cases, it may predict what people prefer better than another person does.

But predicting satisfaction isn't the same as observing my satisfaction with a particular result.

A model can inspect a notification flow and argue that one fewer click is probably better. It can suggest clearer wording or a simpler layout. What it cannot establish from its own output alone is whether the experience matches the product I was picturing.

That isn't meant as an insult to AI. Humans can't directly experience one another's satisfaction either. We ask, observe, test, and revise.

The difference is that autonomous AI workflows can make it very easy to close the feedback loop entirely inside the system. One agent proposes, another judges, and a third optimizes. If the user's experience never enters the loop, the process can become increasingly precise about the wrong target.

So should a human review every iteration?

I don't think so. That would give away much of the value of autonomous agents, and it would confuse two different kinds of feedback.

If a compiler reports an error, I don't need to tell the agent how disappointed I feel about it. If a test proves a contract was broken, the agent can often fix it without involving me. If a deployment policy rejects an unauthorized action, that gate should work whether or not I'm watching.

There are plenty of things AI can evaluate more consistently than a tired human reviewing the twentieth iteration.

There are also situations in which the user doesn't know what they want until they see alternatives. Asking for detailed acceptance criteria up front may simply produce a confident but incomplete description of the desired experience.

For those decisions, I increasingly prefer short, tangible demonstrations. Give me something I can try. Let me react to it. Ask what feels wrong. Then turn that reaction into a clearer architectural or product commitment and let the agents work again.

The point isn't continuous human supervision. It's making sure the feedback comes from a source that can actually judge the property we're trying to improve.

The rabbit hole has a stopping problem

I suspect a lot of agentic rabbit holes are really failures to ask when another iteration is worth doing.

An agent can always find another abstraction to tidy, another test to add, another edge case to protect, or another review to commission. Those improvements may be legitimate. The danger is that the existence of an improvement becomes the justification for doing it.

I've started asking a different question: What will I be able to experience after this change that I couldn't experience before?

For an architecture tool, that might be a diagram I can actually understand. For an approval workflow, an action I can complete without leaving the notification. For a decision-governance feature, a real agent proposing and accepting a decision under delegated authority, with the history and verification limits available for inspection.

These are not the only meaningful milestones. Some foundational work is invisible to users and indispensable. But if we go too long without connecting the work back to an observable outcome, it becomes easy to mistake accumulated engineering activity for progress.

The same question applies to the agents directing other agents: what new evidence will the next iteration produce about whether we're solving the right problem?

The part of the loop that matters

I'm optimistic about increasingly autonomous AI development. I want agents to implement, review, test, and improve software without requiring my permission for every technical choice.

I also think the best agent systems will need to know when their own checks aren't enough. Not because human judgment is always superior, but because some outcomes are defined by the experience of the people the software is for.

A green test suite can establish a great deal. An independent assessment can establish more. Neither should be asked to stand in for a reaction that hasn't happened.

Maybe the most dangerous rabbit hole isn't the one where AI gets lost.

It's the one where AI keeps getting better at something we never stopped to ask whether we wanted.