Field notes · 2026
What 47 people taught us about AI coding agents
We ran a five-week AI pilot across ten departments — not a lab study, a live engineering organisation under delivery pressure. Six findings, with the numbers, including the ones that made me change my mind.
What it added up to
The most durable AI investment is not the tool. It's the context layer — repo-level context files, test harnesses that surface agent mistakes cheaply, recurring workflows checked in as reusable instructions.
And it isn't documentation. Documentation is written for people who fill gaps with judgement. Context files are read by a system that will confidently act on whatever is missing. That makes them infrastructure — with ownership, review, and maintenance.
One line item that attacks the top quality complaint, earns trust for the right reason, survives tool churn, separates high- from low-leverage workflows, and cuts wasted spend.
The six findings
- The most consistent gains weren’t in engineering
Analytics, Marketing, Finance, Customer Service, Supply. By week 5, 88% of non-engineering respondents reported saving 3+ hours a week — higher usefulness, trust, and retention intent than engineering. I expected the opposite.
- Engineering trust plateaued at 3.59 out of 5 — and that’s the right answer
It never crossed 4.0 in any week. For engineers working in real codebases the goal isn’t maximum trust, it’s calibrated trust. Trust moved with task difficulty and observed failure modes, which is exactly what it should do.
- Three evaluations in three years, and the tooling answer kept reversing
Windsurf recommended, Cursor deployed, Claude Code ahead by 2026. Two of the three confident answers I’d have given would now be wrong. The tool is the output; the evaluation instrument is the asset.
- Some engineers saved 10–12 hours a week. Others got almost nothing
Same window, same org, same tools, same codebase. The variance is more useful than the average — and most of the low end was task mismatch, not resistance.
- The ceiling wasn’t model quality. It was usage limits
Weekly limit-hitters went from a quarter to over 40% — half of engineers by the final week. The throttle lands hardest on your heaviest users, who are also your internal advocates. Per-seat planning fails when consumption is this concentrated.
- The #1 complaint tripled over five weeks — the best news in the data
“Wrong assumptions about the codebase” went from 2 of 10 respondents to 6 of 14. Not a model-capability wall but a missing-input problem — which meant my own organisation could fix it. This is where the context-layer argument comes from.
What we're testing next
If the context layer is good enough, does the person driving the agent still need to be the person who knows the codebase best? Same review, same checks, same production controls as any other change — the experiment tests who can drive, not whether engineering stays accountable.
Results and the next series when that pilot wraps.