I Don't Want AI to Just Build What I Ask

I've noticed something interesting about the way I work with AI. Increasingly, the part I value most isn't watching an agent finish a task. It's the conversation we have before either of us knows exactly what the task should be.

There's an appealing model for AI-assisted software development: describe a feature, let the agent work, and inspect the result. Sometimes that's exactly right. If I need a straightforward implementation and can describe the outcome clearly, I don't need a philosophical discussion with my compiler's new best friend.

But some of my most valuable work with AI has happened when I didn't start with a settled answer.

I've come to think of this less as delegating to an unusually fast developer and more as collaborating with another participant in the reasoning. Not because the AI and I are identical, or bear the same responsibility, but because the ideas move in both directions.

An experiment that didn't give me the answer I wanted

I've been building Continuum, a platform for expressing software architecture in a structured way and checking what can be established about its implementation. One of my original assumptions was fairly intuitive: if an AI coding agent follows an explicit architecture, the result ought to be better.

So I tested it.

I had agents build the same application with different degrees of architectural guidance. To my surprise, the agent without Continuum could still make sensible architectural choices and produce a strong implementation. Formal guidance added traceability and exposed some real problems, but it didn't clearly win on finished product quality.

That was an uncomfortable result. A tempting response would have been to refine the experiment until my preferred approach came out ahead, or to argue that we simply hadn't tested the right thing.

Instead, I brought the result into a conversation with my AI collaborator. We kept asking a different question: if strong agents can make good architectural decisions without being told how, what is Continuum actually for?

At first the answer seemed to be architectural visibility before implementation. An agent might design well on its own, but I might still want to understand consequential choices before they become expensive to reverse.

That was useful. We wrote a vision around inspecting architectural intent, verifying implementation, and preserving accountability.

Then the conversation changed the vision again.

The moment the idea stopped being mine

As we worked through an implementation plan, my AI collaborator described a process in which a human would approve architectural decision records before code was written.

It sounded reasonable. It was also narrower than what I wanted.

Why should architectural decisions happen only before implementation? A deployed system changes constantly. A new decision might invalidate part of the existing design, force a migration, or expose a weakness in the original assumptions. And why must a human accept every decision? I might want to delegate some architectural authority to an AI agent, along with validation and verification, while retaining oversight of more consequential changes.

So I challenged the proposal.

We reshaped it around decisions that could be proposed, accepted, revoked, or superseded by authorized humans or AI agents at any point in a system's life. A decision would remain distinct from proof that the architecture or implementation satisfied it. A future iteration could trace that decision into architectural changes, implementation work, and independent re-verification.

This wasn't an AI executing a specification I'd written in advance. Nor was it an AI independently inventing a product while I watched. It was a back-and-forth in which each revision exposed something the previous one had missed.

The resulting idea was stronger because neither side treated the first answer as final.

Autopilot and collaboration solve different problems

Autopilot is enormously useful. Once a problem is understood and the checks are meaningful, I want an agent to have room to work. I don't need to approve every method name, watch every test run, or interrupt a coding session because the agent wants to refactor a helper.

But autopilot assumes we know enough about the desired destination to judge arrival.

In the early part of a project, that assumption is often false. The most important uncertainty may not be how to build the system. It may be which system we should build, what choices we're making implicitly, and which trade-offs are acceptable.

That's where conversation matters. I can offer a product intuition that is difficult to formalize. AI can turn it into a candidate model, point out consequences, suggest alternatives, or ask the question I hadn't thought to ask. I can disagree. The model can revise the argument. We can test the result against actual evidence.

And then, when the direction is clear enough, a coding agent can go implement it.

The distinction isn't that I do the thinking while AI does the typing. Both contribute to the reasoning. The division of labour changes as the problem becomes better understood.

Equal partners doesn't mean interchangeable partners

I described the experience recently as an equal partnership. I still like that description, although it needs a little care.

I'm not claiming the AI has the same experiences, personal interests, or accountability that I do. I live with the consequences of decisions about my products. I decide what I value, which risks I'm willing to accept, and how much authority to delegate. The assistant contributes reasoning and technical analysis, but it can also make mistakes, misunderstand what I mean, or argue confidently from an incomplete picture.

Equality, in the sense I mean it, is about how we treat contributions to the work. An idea shouldn't win just because it came from me. It also shouldn't win because the model stated it fluently. We should be able to challenge both, ask for evidence, and change our minds.

That requires more than friendliness. It requires disagreement.

If every response validates my intuition, we're not doing much collaborative thinking. If I treat every AI suggestion as an instruction, I'm not exercising judgment either. The useful middle is a shared willingness to revise the idea.

There's an obvious limitation: a conversational model may be unusually agreeable, and a compelling discussion can still converge on the wrong conclusion. That's why I keep coming back to experiments, tests, independent checks, and actual user experience. The conversation generates and improves hypotheses; it doesn't certify them.

The implementation is still important

None of this is an argument for endless planning.

In fact, one of the lessons from building Continuum has been that a long implementation phase without anything usable is difficult to evaluate. We've started insisting that each stage deliver a concrete experience.

For the architectural decision work, the first milestone is deliberately modest. An authorized agent should be able to record a proposed decision, accept it within delegated authority, retrieve its history, and show what is or isn't known about alignment and verification. It shouldn't claim to have reconciled an implementation when it hasn't.

That's a small capability compared with the larger ambition. But we can use it, challenge it, and discover what the next stage really needs.

The implementation agent has an important role here. Once we've clarified the desired experience, it can do substantial engineering work without our watching every step. Then the results return to the conversation, where we look at what actually happened rather than simply accepting the success report.

There is no single handoff from human to AI. The process moves between discovery, delegation, evidence, and reconsideration.

The conversation is part of the engineering

I don't know how common this style of working is among other developers. Some may already work this way; others may prefer to delegate complete tasks and inspect finished results. For well-defined work, that approach can be more efficient.

What I do know is that the most consequential changes to my own project recently weren't the result of a particularly clever prompt. They emerged when a conversation made an assumption visible, and one of us challenged it.

I started out trying to make AI follow an architecture. The experiments made me question whether that was enough. Our conversations pushed the idea toward inspectable decisions, delegated authority, and software that can evolve without losing the reasoning behind its design.

I still want AI agents to build things. I want them to take initiative, exercise judgment, and surprise me with solutions I hadn't considered. But I'm no longer interested only in what AI can accomplish on my behalf. I'm interested in what we can discover through the exchange of ideas.

What if the greatest value of AI isn't that it can think for us, but that it challenges us to think with it — not merely as observers of what it creates, but as participants in what we discover together?