# Code Generation Is Fast. Verification Is Not. Agentic IDEs can produce a runnable full-stack application in minutes. That does not mean the application became cheap to trust. In a controlled study of 48 generated projects, the researchers spent as much or more time manually verifying some outputs as the tools spent generating them. Almost half of the recorded errors came from the generated test suites themselves. > The bottleneck is moving from writing code to proving that the code satisfies the requirement. That shift changes how teams should measure coding-agent productivity. Time to first run is useful. It is not the number that decides whether the change is ready to merge. --- ## The experiment tested applications, not isolated functions The forthcoming ASE 2026 paper, [_Generation of Web Apps with Agentic IDEs: An Empirical Assessment_](https://doi.org/10.1145/3832783.3837506), compared GitHub Copilot, Cursor, and Windsurf while holding the underlying model and task specifications largely constant. The tools generated five Python and JavaScript applications from scratch: a note manager, an authentication module, a file host, a real-time chat system, and a distributed task queue. Each condition was repeated three times. The note-manager experiment was later repeated with a newer Claude Sonnet generation, producing 48 project implementations in total. This design matters because a complete application can fail between components even when individual files look reasonable. The researchers checked installation, browser behavior, API behavior, persistence, automated tests, and repository structure. They combined executable checks with manual inspection, then issued corrective prompts when requirements were missed. The study is narrower than production software. It uses five specifications, one primary stack, three IDEs, and limited repetitions. But it tests a more realistic engineering unit than a single function completion: a system whose frontend, backend, storage, and tests have to agree. --- ## Familiar patterns compressed cleanly; coordination did not CRUD and authentication produced relatively few errors. The real-time chat and task-queue applications were less reliable because they required concurrency, asynchronous workers, retries, cancellation, persistence, and multi-component coordination. The task queue recorded the highest backend reliability and maintainability issue counts across all three IDEs. Its generated backend test coverage ranged from 60.7% to 68.1%. Coverage alone does not prove test quality, but here it reinforces the larger result: the architecture with the most coordination semantics was also the hardest for the tools to implement and verify. That pattern is more useful than a tool ranking. Most runs used Claude Sonnet 3.5, and only the note manager was repeated with Sonnet 4.5. The absolute ordering of Copilot, Cursor, and Windsurf is already time-sensitive. The architectural gradient is more durable. Agents handled established request-response patterns better than stateful, distributed behavior. A team can use that distinction before it has enough internal data to estimate a precise defect rate. --- ## The generated tests were not an independent judge Automated test-suite generation accounted for **49.6% of all observed errors** in the study. The failures clustered around complex I/O, asynchronous behavior, and edge cases. This exposes a circularity that a green test run can hide. If one agent interprets the requirement, writes the implementation, and derives the tests from the same interpretation, the implementation and tests can agree while both miss the intended behavior. The paper did not run an experiment comparing self-generated and independently authored acceptance suites. So the next point is my engineering inference, not a result the authors measured: acceptance criteria should be owned outside the generation loop. That can mean deterministic tests written before the task, a separate reviewer translating the requirement into checks, or an evaluation harness the coding agent cannot edit. The mechanism matters more than whether a second model is involved. Independence comes from separate authority over the acceptance contract, not a different chat window. --- ## The strongest case for agentic IDEs still holds The obvious counterargument is that manual verification is not new. Engineers already review, test, and debug human-written code. If an agent produces a first implementation in minutes, even a substantial review pass may still reduce total delivery time. The data supports that case. Average agent execution time ranged from 6:33 for Cursor to 14:07 for Windsurf. These are meaningful compressions for full-stack scaffolding. The newer-model note-manager runs also reduced both generation and verification time substantially, suggesting that the tools are improving. The mistake is not using them. The mistake is crediting the entire generation-time reduction as delivery-time reduction. Manual verification averaged 11:40 for Cursor, 12:22 for Copilot, and 12:24 for Windsurf. For Copilot and Cursor, the average inspection time exceeded generation time. The task queue required roughly 14 to 19 minutes of verification depending on the IDE. Multiple corrective rounds were common, and fixes introduced regressions in some note-manager and task-queue runs. The faster the first draft becomes, the larger verification looks as a share of the workflow. That is not proof that agents fail. It is evidence that the productivity metric has to move downstream. --- ## Measure accepted changes, not generated code Lines generated and time to first run reward visible activity. They do not price specification work, acceptance testing, architecture review, regression repair, or escaped defects. A more useful rollout scorecard separates work by architecture type and records: | Measure | What it reveals | | ------------------------------------------ | ---------------------------- | | Time to first runnable result | Generation speed | | Reviewer minutes per accepted change | Human verification cost | | Corrective rounds and regressions | Stability of the repair loop | | Acceptance failures after agent tests pass | Self-validation gaps | | Escaped defects by system boundary | Production risk | This also suggests a risk-based operating rule. Use agents aggressively for familiar, bounded patterns. Raise the strength of independent tests, observability, and human review as asynchronous state and component boundaries increase. That rule is deliberately conditional. The study does not establish a universal failure rate, and its manual classifications involve evaluator judgment. A React application, a different model family, or a repository-modification task may behave differently. But the decision does not require pretending the evidence is broader than it is. The paper establishes that fast generation and cheap verification are separate properties. A production team should measure both. --- ## The verdict Coding agents are already useful at compressing familiar implementation work. They have not removed verification work. In complex systems, they can concentrate it around the exact boundaries that are easiest to miss: asynchronous behavior, shared state, and generated tests that validate the agent's own interpretation. The winning team will not be the one that generates the most code. It will be the one that can prove, with the least independent reviewer effort, that the generated change is safe to accept.