Over the last year, my day-to-day job has changed in a way I am still trying to understand. I am still an engineer. I still design systems, read code, debug failures, review implementations, and sometimes build things myself. But I write much less code than I used to, and that feels weird.
For most of my career, producing software was the visible evidence that I was doing my job. You had a problem, you designed a solution, you wrote the code, and something that did not exist in the morning existed by the end of the day. There was a very direct relationship between effort and output. Today, that relationship is disappearing.
A large part of the implementation work around me can now be generated by AI. Not perfectly, not autonomously, and not without supervision, but cheaply enough that the first implementation is increasingly not the hard part. If I need an API endpoint, a migration, a data transformation, a test suite, an integration, or some internal tooling, I can describe the problem, provide enough context, and get a plausible implementation surprisingly quickly.
The expensive part starts afterwards: does it actually work?
Not simply, "does the code run?" or "do the tests pass?" The real question is whether the feature behaves correctly across the range of situations we actually care about. That question has slowly become a much larger part of my job, and it has changed how I think about my own value.
A year ago, I spent much more time thinking about how to implement something. Now I spend much more time thinking about how it can fail. What happens when the model receives something we did not anticipate? When two components disagree? When an apparently good answer contains the wrong evidence? When the evaluator itself is biased? What happens when a model succeeds 95 percent of the time, but the remaining 5 percent contains exactly the failures that matter to the business? What happens when a fallback silently changes provider, region, latency, cost, or behavior? What happens when everything returns HTTP 200 and the system is still wrong?
The implementation is often no longer the difficult intellectual problem. Failure detection is. Failure prevention is. Defining what "good enough to ship" actually means is.
That creates a strange inversion. For decades, software engineering treated testing and validation as something downstream of implementation. First you built the thing, then someone checked whether it worked. In AI systems, I increasingly feel the opposite. Generating the implementation is becoming cheaper, while knowing whether you should trust it is becoming more expensive.
This week, for example, while I am taking some time off, the team asked me for a new evaluator for a feature. At first glance, that sounds almost like QA work, and I admit that part of me reacts negatively to that idea. I spent years learning how to build software. Am I slowly becoming the person who checks everybody else's work?
But I think that interpretation misses what is actually happening.
The hard part of building an evaluator is rarely writing the evaluator. AI can help enormously with that too. The difficult part is deciding what the evaluator should measure in the first place. What counts as failure? Which failures can be checked deterministically? Which ones require statistical evaluation? Which dataset represents reality well enough? Where are the blind spots? Can the evaluator itself be fooled? Are two supposedly independent checks actually making the same mistake? What threshold is sufficient to release something to production?
Those are not really questions about test implementation. They are questions about the operational definition of correctness. And with AI, correctness is becoming something we have to engineer.
Traditional software gives us a relatively comfortable contract. For the same inputs and state, deterministic software should generally produce the same output. LLMs break that assumption. They generate plausible outputs across enormous input spaces, and they can produce something structurally correct, linguistically excellent, internally coherent, and completely wrong.
So we wrap them in software. We constrain them, validate their outputs, compare signals, build fallback paths, add deterministic checks around probabilistic behavior, create evaluation datasets, measure regressions, and observe production traces. Then, inevitably, we discover a failure mode we did not know existed and modify the system again.
A surprising amount of my work now lives in that boundary. Not creating intelligence, but making probabilistic intelligence reliable enough to become part of a real product.
This also makes me uncomfortable for another reason. If AI can generate the implementation, why couldn't another AI eventually generate the evaluations too? Why couldn't I describe my principles, my distrust, my way of looking for edge cases, and encode all of that into another agent?
The answer is probably that I can. And I should.
If I discover a reliable failure pattern, I want to automate its detection. If I repeatedly perform the same review, I want a system to perform it for me. If a deterministic gate can replace my manual judgment, building that gate is progress. My value cannot depend on protecting tasks from automation, because that would be a losing strategy. So perhaps the important distinction is not between work humans can do and work AI can do. It is between known problems and unknown ones.
Once a failure mode is understood, it becomes cheaper. We can encode it in a test, build an evaluator, add a policy, or teach an agent to look for it. The difficult part then moves somewhere else. The frontier becomes the next thing we do not yet know how to measure, and that seems to be where more and more of my work is going.
My output is increasingly not code. It is a definition, a constraint, an architecture, a failure taxonomy, a release criterion, or a deterministic guard around something probabilistic. Sometimes the final artifact is twenty lines of code, but those twenty lines may represent two days of thinking about what exactly needs to be prevented.
That changes how productivity feels. If I generated 1,000 lines of production code in a week, I could easily point at what I created. If I spend the same week investigating one subtle failure mode and eventually add a tiny check that prevents it, the visible output looks much smaller. But the economic value might be significantly larger.
The system already knows how to generate. The harder question is whether we can depend on what it generates.
I do not know exactly what this role should be called yet. AI engineer still fits. Reliability engineer fits part of it. Architecture fits another part. Evaluation engineering is clearly becoming important. None of those labels completely captures the transition I am experiencing.
What I do know is that I am moving away from being primarily the person who produces the implementation. I am becoming the person who asks what the implementation must prove before we trust it.
That feels strange because software engineering trained us to identify ourselves with building. Code was craftsmanship, output, and evidence that we were useful. AI is making code abundant, and when something becomes abundant, value usually moves somewhere else.
Maybe the next scarce resource in software engineering is not implementation. Maybe it is the ability to determine when an implementation is wrong before reality does it for you.
The cheaper part is increasingly generated. The expensive part is knowing where it will fail, and making sure it doesn't.
Human concept, nice written by AI.