The Right to See What AI Is Building

As autonomous agents become capable of building complete software systems, architectural visibility may become more important than architectural control.

I've been questioning something about the role of software architecture in an age of autonomous AI.

For most of my career, architecture has been something we do to guide implementation. We examine requirements, identify boundaries, define responsibilities, make technology decisions, and establish the constraints that developers are expected to respect.

The assumption is that good architecture helps us build better software.

But I've been experimenting with autonomous coding agents, and some of the results have challenged that assumption.

In one experiment, I asked two AI agents to build the same application. One had access to a formal architecture system that could model the design, bind it to technology choices, and verify its implementation. The other worked without it.

Both produced remarkably similar results. Both made sound architectural decisions. Both built polished, working applications. Both successfully handled a significant change to the original requirements.

The architecture tooling produced better traceability and caught a real structural problem. But it didn't produce noticeably better software.

At first, that was disappointing.

Then I realized I might have been measuring the wrong thing.

The Architecture That Didn't Need an Architect

The interesting thing about the experiment wasn't that AI could write code. We've known that for a while.

It was that the agents made architectural decisions I would normally expect from experienced software engineers.

They understood ownership boundaries. They designed database constraints to handle concurrent operations. They separated business responsibilities. They recognized which existing rules needed to survive a new feature.

The agent working without formal architecture guidance didn't produce an architectural mess. In some respects, it made better product decisions than the guided agent.

That raises an uncomfortable question for someone who has spent considerable time building tools to formalize software architecture.

If the AI can design a good architecture on its own, why should we tell it how to architect the system?

One answer is that we shouldn't.

At least, not automatically.

An expert developer doesn't become more effective simply because someone hands them a longer list of instructions. Sometimes additional constraints prevent them from finding a better solution.

AI agents may be no different.

But there's something important that this conclusion overlooks.

What If I Want to See the Architecture First?

Imagine asking an AI agent to build an application.

You explain the product you want, describe its users, and identify the important behaviours. The agent begins working. It makes hundreds of decisions about structure, data, technology, communication, security, and deployment.

A few hours later, you have a working application.

It's impressive. It may even be exactly what you wanted.

But what did the agent decide?

Where does the data live? Which component owns each responsibility? What happens when an operation fails halfway through? Which assumptions are embedded in the design? What will be difficult to change later?

You can ask the agent to explain the architecture afterward.

It might give you an excellent explanation. It can inspect the code, reconstruct the design, and draw a diagram.

But that's different from having had the opportunity to inspect the architecture before the system was built.

By then, the decisions have already been implemented. Revising them may involve changing a considerable amount of working software.

This is where I began to see a different value in formal architecture.

Not necessarily making the AI a better architect.

Making its architecture visible to me before I commit to it.

Visibility Is Not the Same as Control

There's an important distinction here.

If I ask an agent to describe its proposed architecture, I'm not necessarily telling it what to build. I'm asking it to expose the decisions it's considering.

That distinction matters because the goal shouldn't be to prevent the agent from exercising judgment.

I want an AI developer that can challenge my assumptions. I want it to recognize better solutions than the ones I initially imagined. I want it to explain why a proposed boundary is unnecessary, or why a technology choice introduces risk.

I don't want to replace that reasoning with a rigid checklist.

What I want is a way to participate in the decisions that matter.

Some architectural choices are preferences. Others are commitments.

Choosing how to organize internal modules might be something the agent can decide freely. Deciding that customer information must remain within a particular security boundary is a different kind of decision.

An architecture system should help distinguish the two.

The agent should be able to propose changes, explain trade-offs, and revise its design as it learns. But when a decision protects something important, there should be a record of what was agreed and a way to determine whether the implementation honors it.

That isn't micromanagement.

It's informed delegation.

The Difference Between a Diagram and a Commitment

We've had architecture diagrams and design documents for decades.

They can be useful, but they have a familiar problem: there's no guarantee the implementation follows them.

A developer can produce a beautiful design, receive approval, and then make perfectly reasonable changes during implementation. Eventually the diagram describes something that no longer exists.

Sometimes those changes are improvements. Sometimes they're accidental deviations. Often nobody can tell the difference without investigating the code.

I've been working on Continuum, a system for expressing architecture in a standardized semantic language and connecting it to implementation evidence.

One of the things it allows is inspecting the proposed architecture before implementation. Capabilities, responsibilities, relationships, technology bindings, and constraints can be examined as explicit elements rather than inferred from a collection of diagrams.

More importantly, those elements can remain part of the development process.

In the experiment, the application built with Continuum showed strong alignment with its original architectural model. Its requirements were traced to tests, its implementation was checked against the declared structure, and its architecture was available for inspection throughout development.

That doesn't mean we formally proved every aspect of conformance. Some verification remained incomplete, including an architectural change blocked by a tooling defect.

But the underlying distinction is important.

A diagram communicates an idea.

An inspectable, verifiable architectural model creates the possibility of maintaining an agreement.

And when an agent needs to depart from that agreement, the interesting question isn't simply whether it violated a rule. It's why the agreement should change.

The Part We Still Got Wrong

The experiment revealed another problem.

Both agents implemented a lending application where people could reserve physical items. Both correctly prevented overlapping reservations based on scheduled dates.

Both passed their tests.

But both missed the same scenario.

If someone borrowed an item and failed to return it on time, the system could approve another reservation even though the original borrower still possessed the item.

The date calculations were correct. The database constraints worked. The architecture was internally consistent.

The assumption was wrong.

A scheduled return date was being treated as evidence of actual availability.

This is the limitation that makes architectural visibility only the beginning of the story.

Inspecting a model doesn't guarantee that we've modelled the right things. Formal verification can establish that a system obeys its declared rules without establishing that the rules adequately describe reality.

The missing requirement wasn't simply better code.

It was a challenge to the judgment behind the code.

Why had availability been defined in terms of scheduled dates? What evidence justified that assumption? What should happen when physical custody and reservation schedules disagree?

Those questions belong to architecture too.

But they're questions about the reasoning that produced the architecture, not merely its structure.

Architecture as a Record of Judgment

This has changed how I think about the purpose of architectural tooling.

I used to think primarily about making architecture executable: creating a model that an agent could follow and that verification tools could enforce.

I still see value in that.

But I'm beginning to think the more important opportunity is making architectural judgment inspectable and accountable.

That would mean preserving more than components and dependencies.

For consequential decisions, we would also preserve the rationale, assumptions, alternatives, supporting evidence, and the conditions under which the decision should be reconsidered.

We would know not just that a reservation service uses date-based conflict checks, but why that rule was chosen and what assumptions it depends upon.

When new evidence contradicts those assumptions, the architecture could be challenged rather than silently treated as authoritative.

And when the system evolves, we could determine whether an earlier decision remains justified.

This becomes particularly interesting when the developers are autonomous agents.

The agent that implemented the software should not necessarily be the only authority evaluating its decisions. Independent review, deterministic verification, operational evidence, and human judgment should all have a place.

Otherwise, we risk creating an elaborate system where AI writes the design, implements it, generates the tests, and then certifies that everything agrees with what it originally decided.

That's consistency.

It isn't necessarily accountability.

The Human Doesn't Have to Hold Every Control

I keep returning to the analogy of modern aviation.

Pilots don't manually control every movement of a commercial aircraft. Automation performs much of the routine work, often with extraordinary precision.

But the crew's ability to understand the aircraft's state, anticipate problems, and intervene remains essential to the safety system.

The value isn't simply that a human can take the controls.

It's that the human has meaningful situational awareness and the authority to act.

I think autonomous software development needs its own version of that visibility.

Not because humans will always make better technical decisions than AI. We may already be approaching situations where that isn't true.

And not because every application needs an elaborate approval process.

But because delegating an expert decision doesn't eliminate the need to understand its consequences.

For people without years of software engineering experience, this could be especially important.

They may soon be able to build sophisticated applications without knowing how to implement them. That is an extraordinary opportunity.

But there's a difference between being able to commission software and being able to exercise meaningful ownership over it.

If all the important decisions are hidden inside an autonomous development process, the owner may receive a working product without understanding the system they've become responsible for.

Making those decisions visible, understandable, and reviewable could help close that gap.

Of course, a detailed architecture model isn't automatically understandable to a non-technical person. The presentation matters as much as the underlying formalism. A meaningful review needs explanations of consequences, not just more sophisticated boxes and arrows.

And we still need evidence that such reviews lead to better decisions. Visibility is a capability, not proof of value by itself.

What Architecture Might Become

I don't think the future of software architecture is necessarily architects prescribing how AI agents should build systems.

The agents may not need that much guidance.

Instead, I see three responsibilities becoming increasingly important.

Before implementation, make the proposed architecture inspectable so important decisions can be questioned and accepted.

During implementation, establish whether the software remains faithful to those decisions, while allowing justified changes.

After implementation, preserve the reasoning and evidence so decisions can be challenged as requirements, knowledge, and operating conditions evolve.

Inspect. Verify. Account.

None of those requires the human to be a better programmer than the machine.

They require us to distinguish between delegating the work and surrendering our ability to understand it.

That's the part of the automation discussion I find increasingly important.

We may reach a point where AI builds software more effectively than most human teams. Perhaps we'll even come to trust its engineering decisions more than our own.

If that happens, forcing humans to direct every architectural choice may become an unnecessary constraint.

But allowing software to be built without any meaningful record of why it was built that way creates a different problem.

The future of architecture may be less about telling intelligent systems what to do, and more about making their decisions understandable, verifiable, and open to challenge.

We don't have to make every decision ourselves. But we should be able to see what we've agreed to.