More than one in five code reviews on GitHub now involve an agent, and Copilot code review alone has processed over 60 million reviews in under a year of 10x growth, according to GitHub. If you're commissioning custom software right now, that number should change the questions you ask your vendor.
The pull request has become the unit of work an AI agent produces. An agent picks up an issue, drafts a plan, writes the code on a branch, runs the tests, and hands a human the diff. Programming Insider's recent coverage of how agentic coding assistants are reshaping software delivery frames the shift bluntly: the prompt window has moved into the PR, and everything that governs merges, including review, CI, and branch protection, is suddenly the real throughput constraint. Buyers who used to ask "how fast can you ship?" need a different set of questions now.
Decide Whether Your Vendor's Agent Is Writing or Just Suggesting
Two very different setups get described with the same words. One is autocomplete that lives in the editor and offers suggestions a developer accepts a line at a time. The other is a background agent that picks up a ticket, works on a branch, and submits a pull request as if it were a teammate. Only the second one changes who does what on your project.
Ask your vendor which model they're using, where the agent is allowed to run, and what it has permission to touch. If an agent is opening PRs against your repository, the useful follow-ups are concrete: Which entry points can it be triggered from? What credentials and secrets can it reach?
Who approves its first run of the day? Vague answers here are the warning sign.
Decide What Counts as 'Done' Before You Accept Volume as Progress
More PRs is not the same as more delivered software. An agent can comfortably open ten pull requests in the time a senior engineer would open two, and that can look like acceleration right up until the review queue stops moving. The bottleneck slides from writing to reading.
Agree on definitions with your vendor before the first sprint. A PR is "done" when a qualified human has reviewed the diff, the test suite covers the new behavior, and the change has survived whatever staging checks you've agreed on, not when the agent marks the task complete. If weekly reporting is going to count merged PRs, make sure it also counts rework, revert rate, and reviewer hours spent.
Decide Who Is Accountable for the Review
When an agent submits code through the same door as a junior engineer, the review becomes the governing act of the whole pipeline. That's a different job than skimming a diff written by someone whose reasoning you already trust. Agent-generated changes tend to be verbose, prone to silent duplication, and surprisingly comfortable to approve, which is the trap.
The questions worth asking a vendor:
- Named reviewer. Which human engineer signs off on agent PRs touching your codebase, and what is their seniority? "The team reviews it" is not an answer.
- Review budget. How many hours a week are allocated to review, and does that scale with the number of PRs the agent opens?
- Rejection path. What happens when a PR is wrong, and how is that fed back into the agent's prompts or instructions so the same mistake doesn't recur in the next ticket?
Decide How CI and Tests Earn Their Keep
If an agent can write code faster than humans can read it, your continuous integration pipeline is now a safety net you're betting the project on. Flaky tests, missing coverage, and slow build times stop being annoyances and start being what lets bad changes slip into main. The investment case for shoring up CI changes the moment agent PRs enter the queue.
Ask your vendor what their test coverage targets are for agent-authored changes specifically, how long the full pipeline takes to run, and what secondary checks, including static analysis, dependency scanning, and secrets detection, run before a human is asked to look. For AI-generated code in particular, NIST's Secure Software Development Framework is the standard to point to when you want a vendor to show their work on how model-produced code is governed through the SDLC.
The Buyer's Job Has Shifted Up One Level
The old buyer's instinct was to pick a vendor and let them manage the engineering. That still works, but the thing you're actually buying has changed shape. You are buying a review pipeline with people and agents feeding into it, and the quality of what comes out the other end depends on how that pipeline is run.
Ask about it the way you'd ask about staffing, timelines, and price. It's the same conversation now.
