Who Takes the Controls?

I've been thinking about the pilot sitting in the cockpit of a modern airliner.

For much of a commercial flight, automated systems can manage the aircraft's path with extraordinary precision. The pilots aren't just passengers. They monitor, communicate, anticipate, decide, and intervene. But the relationship between the expert and the machine has changed. The person with the greatest responsibility is no longer necessarily the one performing most of the routine work.

That reminded me of something I recently saw in software engineering.

I gave two AI coding agents the same product vision: build a small application that helps people lend belongings to one another. One agent had access to a formal architectural modelling and verification system. The other worked independently, using code, tests, and ordinary engineering documents. Both built complete applications. Both handled a substantial follow-up change. Both produced software that was remarkably close in quality.

I had expected the architecture tools to make a more noticeable difference. Instead, I found myself questioning something larger than architecture.

If an AI can do the work of an expert, what is the expert for?

The expert who isn't holding the controls

Autopilot is sometimes described as though the human pilot is there mainly to make passengers comfortable. I understand the intuition. We trust an aircraft differently when we know trained people are up front.

But that's an incomplete picture. Pilots do far more than reassure passengers, and an autopilot is not an autonomous pilot. It operates within capabilities and procedures, while the crew remains responsible for managing the flight. Aviation safety comes from the whole system: aircraft design, training, operational discipline, crew judgment, redundancy, and learning from failures.

Still, the analogy reveals a genuine paradox.

The better the automation becomes at routine work, the less often the human needs to perform that work. Yet when the automation encounters something unusual, we expect the human to recognize the problem and respond expertly.

How does someone remain excellent at a skill they rarely get to practise?

Aviation takes that question seriously. Transport Canada's guidance on flight-deck automation warns that continuous use of automated flight systems may degrade manual skills, and it calls for opportunities to maintain proficiency. That is a real operational problem, not merely a philosophical one.

Now imagine software development reaching a similar point.

The software built itself. Almost.

In my experiment, both coding agents implemented the same user journeys, enforced important business rules, handled concurrent requests, and passed the test suites. When asked to add lending circles, both preserved existing reservations even when a person's membership changed.

That was an important design decision. Membership should determine whether someone is allowed to request an item. It should not retroactively erase a commitment that was legitimate when it was made.

Both agents arrived at it.

The formal architecture system supplied stronger traceability and even caught a real structural problem. Those are useful contributions. But it didn't yield a noticeably better product. The independently working agent did at least as well in this one experiment.

The result doesn't establish that architecture has become unnecessary, or that every agent can perform this way on every project. It does challenge a comfortable assumption: that an expert human must continually prescribe how software is built in order for the result to be good.

And then the reviewers found a bug both implementations had missed.

An overdue item could still be out in someone's hands while the system approved a future booking based on its scheduled return date. The code's date-range checks were internally consistent. The tests passed. But the real world doesn't return a physical object just because a calendar says it's due.

The failure wasn't primarily a lack of programming skill. It was a missing understanding of the situation the program was supposed to represent.

A human could have made the same mistake. That isn't the point. The point is that someone—or something—had to question an assumption the entire implementation had accepted.

What happens to the value of expertise?

I've been thinking about this in two ways.

The first is economic. If one person with capable AI tools can produce what previously required a team, some kinds of engineering labour will become less scarce. The value of routine implementation may fall, even as the amount of software being built increases. It would be comforting to say that human experts will simply move up a level and do all the judgment, architecture, and product thinking.

But I'm not sure we can assume those activities will remain uniquely human either.

The experiment was unsettling precisely because the agent wasn't merely typing code. It made architectural decisions, chose data boundaries, reasoned about concurrency, built interfaces, and revised its own work. The distinction between execution and expertise is becoming harder to draw.

The second question is about capability itself.

Traditional expertise is developed through doing. We acquire judgment by making decisions, seeing consequences, correcting mistakes, and gradually recognizing patterns before they become problems. The junior developer who struggles with a race condition today may become the architect who recognizes a dangerous state transition years later.

What happens to that learning path when the machine solves the race condition before the developer has even noticed it?

We may create a generation of people expected to supervise systems they have had fewer opportunities to understand from first principles.

Or perhaps AI will teach those principles more effectively than the old apprenticeship model ever did. That's possible too. A good automated tutor could expose more people to difficult decisions, explain alternatives, and simulate failures that would otherwise take years to encounter.

Automation does not have to destroy expertise. But neither does it automatically preserve it.

Reassurance is not the same as responsibility

There's another uncomfortable part of the aviation analogy.

A human in the cockpit is visible. We know whom to ask what happened. After a serious incident, investigators can examine crew decisions alongside procedures, equipment, organizational failures, and system design. The presence of people does not make them the sole cause of failure—or the sole source of safety.

With autonomous software, accountability becomes harder to picture.

Suppose an agent designs a system, writes it, tests it, deploys it, and then monitors it. The owner sees green checks and a polished interface. When something goes wrong, we can collect logs, inspect code, reproduce the fault, and ask the agent for an explanation.

But who had enough independent understanding to know that the checks were missing something?

A named human approver doesn't solve this merely by being named. If the person cannot meaningfully evaluate what they are approving, the approval may be more comforting than protective.

This is where I find myself returning to architecture, but for a different reason than before.

Maybe its future value isn't telling a capable agent how to build software. Maybe it's preserving enough knowledge about the system that people and other agents can independently examine its assumptions: what the software is supposed to guarantee, why decisions were made, which evidence supports those decisions, and what remains uncertain.

That record would have to be useful, not just formal. A beautifully documented mistake is still a mistake. And independent AI verification may eventually outperform human review in many situations. We shouldn't assume human oversight is valuable merely because a human performed it.

We should ask what the oversight can actually detect.

The part I can't resolve

It's tempting to end with the familiar reassurance that AI will handle the repetitive work while humans supply creativity, judgment, and purpose.

I hope that happens. I don't think we know that it will.

AI is already participating in activities we once used to define expertise. Its economic value may come partly from replacing work that humans spent years learning to do. And if it becomes better at making difficult decisions as well as executing them, the market may not reward human judgment simply because it is human.

There are still choices about goals, acceptable risk, and whose interests a system should serve. People and institutions remain responsible for those choices. But responsibility does not guarantee that the person bearing it will have the competence or opportunity to exercise meaningful control.

That, to me, is the deeper irony.

We're building machines to relieve us of difficult work. Much of that work is also how we learn to recognize when something is wrong.

If we succeed, we may need to become far more deliberate about how expertise is developed, preserved, and exercised. Not because humans must always outperform machines, but because a society that delegates its judgment still needs a way to examine the judgments made on its behalf.

The question isn't only whether the autopilot can fly.

It's whether someone will still understand the flight well enough to know when it shouldn't.