In Part 1, I said the highest-leverage investment we made was giving agents somewhere to validate themselves. A few people asked what that actually looks like.

This post is the answer. It covers the engineering side: six problems that stood between us and a pipeline producing work we’d actually merge, in the order we hit them, and what we built for each.

Problem #1: We were the blocker

The whole point of an agentic pipeline is that work moves without someone standing over it. An engineer approves a ticket in the evening, and by morning there’s a pull request waiting, already tested and verified. That only works if nothing in the middle of the run needs us.

Every ambiguity in a ticket breaks that. A human engineer can ask in Slack and wait. An agent either stalls or, worse, confidently picks the wrong answer. Either way, we’re back in the loop.

So we gave the agent a place to ask questions before any code exists.

The first stage of the pipeline is an agent that reads the ticket and the relevant code, then writes a short spec: what’s decided, what’s changed since the ticket was written, how we’ll know it’s done, and where the change lands. It covers the what and why, with acceptance criteria. It never covers the how.

Every open question comes up here, gets answered once, and gets written down. A human settles anything that’s still undecided and approves the spec. From there, the coding agent has a brief it can execute end to end. If something still isn’t decided, the run stops at the spec instead of burning a development cycle.

We enforce the what/how line harder than any other rule. Once a spec starts dictating implementation, you end up reviewing the same decision twice, once in the spec and again in the diff. The spec is the contract. How to meet it is the coding agent’s job.

The difference showed up fast. Tickets written the old way triggered several blocking questions from the spec agent. Tickets written through our product interview skill triggered none.

That pointed us upstream. The spec stage keeps questions from interrupting development, but the questions still exist. A well-written ticket doesn’t raise them in the first place, and eventually won’t need a human at the spec gate at all. So we moved some of our investment from the coding stage to how tickets get written. More on that in Part 3.

Problem #2: An agent that can’t run its code is guessing

For most of what we ship, unit tests aren’t real verification. Our services talk to databases, message brokers, object storage, and mail paths. A detector that passes in isolation tells you very little about whether a message actually made it through the system and landed where it should.

So we gave the agent an environment where it can spin up the services it needs, run the real flow, watch what happens, fix what breaks, and verify the result end to end.

Our first unattended run took 21 minutes, 103 turns, and $7.92, and the agent didn’t ask a single question. It wrote a detector, watched its own test fail, fixed the fixture, ran it again, and confirmed the result by reading back the rows it had persisted. Then it put that evidence in the pull request.

That’s the bar. An agent that can make a change, verify it, and show us the proof, with nobody watching.

Problem #3: The runner and the laptop have to be the same machine

Self-verification raised a new problem: we had to trust the environment the agent was verifying against.

The failure mode is predictable. The agent’s environment slowly drifts from the engineers’ until one day the agent is verifying something nobody actually ships. Let that happen a few times and nobody trusts the pipeline anymore.

So we made the environment code instead of configuration. The stack is defined as named bundles in a file that lives next to the code, and one script boots any of them. An engineer on a laptop and an agent on a runner use the same command against the same definition.

We applied the same rule to the agent itself. Its configuration, skills, permissions, and hooks live in the repo too, so changing how an agent behaves goes through pull request review like any other change. No runner is quietly carrying a setting someone tweaked in a terminal eight weeks ago.

If we’re going to trust an agent’s verification, the environment behind it has to be reproducible, versioned, and the same one our engineers use.

Problem #4. A reviewer shouldn’t have to check out a branch to see what changed

Once agents write the code, review is pretty much the whole job. And review turned out to be two jobs with one name. The mechanical half asks whether it works, and that’s now the agent’s problem, answered before a pull request even exists. The judgment half asks whether it’s right. That’s the only part we ask a human for, so everything else is about getting them there faster, with everything they need.

Take an interface change. To review one properly, someone used to have to pull the branch, run the app, find the affected flow, and click through it before they could even start judging the change. That’s the kind of friction that turns a two-minute review into tomorrow’s problem.

Now, when a change touches the interface, the agent captures every state it affected, records the full click-through, and posts both on the pull request. The reviewer sees what changed without running anything.

By the time a pull request reaches a human, everything they need to review it should already be there.

Problem #5. Not every pull request deserves the same attention

The more code agents produce, the more valuable human attention gets. A small, isolated change shouldn’t get the same scrutiny as one touching authentication.

So before a human opens a pull request, we run a risk assessment and score it from one to ten. The score combines signals like blast radius, high-risk paths, missing test coverage, feature flags, and code ownership, plus a model pass that explains the risk and tells the reviewer where to look first.

On one change, it caught an open redirect in an authentication flow from the diff alone, on a branch no test covered. That’s what it’s for: pointing human judgment at the places it matters most.

The score is advisory and never blocks a merge. That’s on purpose. Engineers learn to work around a blocking score. A score that’s consistently useful is one they learn to trust.

Problem #. We were running on feel

We built the system first and the measurement second. I wouldn’t do it in that order again.

Once the pipeline was running, we needed to see where work was moving, where it got stuck, and why. So every stage became a label on the ticket, with its timestamp kept. Every agent session is instrumented too: cost, tokens, duration, tools called, errors, permission prompts, repo, pull request, and runner.

That changed how we improved the system. Our coding stage, for example, looked like it took 41 minutes. The agent was actually coding for 18. The rest was waiting for a free runner. Without the data, we would have optimized the wrong thing.

It also changed how we drove adoption. We could see which teams were using the pipeline, where people dropped out of the flow, and which stages caused friction. Instead of wondering why adoption wasn’t growing, we could see exactly where it broke and fix that.

The infrastructure took months to build. Getting people to change how they work doesn’t have an end date. The dashboard ended up being less about measuring the pipeline and more about how we keep improving it and bring more of the company along.

Where to start

1. Start upstream. The better the definition, the fewer decisions an agent has to bring back to a human.

2. Give agents more than a place to write code. They should be able to run the real system, test their work, fix what fails, and prove the result end to end.

3. Make that environment reproducible. Engineers and agents should work from the same definitions, and the agent’s skills, permissions, and hooks should go through review like everything else.

4. Design for human judgment, not human verification. By the time a pull request reaches an engineer, the mechanical work should be done, the evidence should be attached, and the reviewer should know where to look.

5. Measure from day one. Instrument every stage so you can separate waiting from working, find bottlenecks, and see where adoption breaks.

We didn’t take engineers out of engineering. We changed where their time goes. Agents handle more of the execution, testing, and verification. Engineers bring the context, the decisions, and the judgment, which is the part humans are actually best at.