← Back to blog
13 min read

Code Review in the Age of AI: You Don't Get to Skip the Lines

Popular advice says to review the intent and stop reading the lines. The lines are where agent-written code goes wrong: N+1 queries, swallowed errors, and tests written by the same model that wrote the bug.

#ai#code-review#engineering-practice

The case for reviewing only the intent

You have probably heard this argument a lot recently: agents write most of the code, there is too much of it for humans to read, so line-by-line review is outdated. Review the intent instead (the spec, the interfaces, the test names) and treat the implementation like compiler output. Nobody reviews the assembly their compiler produces, so why review the Kotlin an agent produces?

The sources people quote for it are more careful than the argument. Andrej Karpathy’s vibe coding tweet (“I ‘Accept All’ always, I don’t read the diffs anymore”) describes how he builds throwaway weekend projects, not how he ships software, and he says so in the same post. Simon Willison, quoting it, draws the line: if you reviewed the code an LLM wrote, tested it, and can explain how it works, “that’s not vibe coding, it’s software development.” GitHub’s spec-driven development toolkit says “specs become the shared source of truth”, and the same post tells developers that their role “isn’t just to steer. It’s to verify.” Addy Osmani’s piece on code review in the age of AI says AI is moving review “from line-by-line gatekeeping into higher-level quality control”, and finishes the sentence with “human judgment remains the safety-critical component.” The intent-only position is what you get when you keep only the first half of each statement.

An agent writes hundreds of lines diff in minutes and a reviewer reads a few hundred lines an hour, so not reviewing the lines is a natural position in an attempt to speed up the process but doesn’t remove the bugs; you just stop seeing them.

Agent code looks correct even when it isn’t

The intent-only argument depends on the compiler comparison, which is faulty. A compiler is deterministic and has a contract: if the source is correct, the output is correct, apart from rare compiler bugs. An LLM is not. It produces the code that looks like a solution. Often acceptable solution, but not always, and you can’t tell the difference by only relying on intent.

This breaks a rule reviewers have relied on for decades: code that looks sloppy usually is sloppy. Bad human-written code tends to show it through inconsistent names, odd structure, and commented-out experiments. Agent-written code is always well formatted, well named, commented, and green in CI, whatever the quality underneath. A senior engineer used to skim a diff in ninety seconds and spot the trouble. Now the skim says “fine” every time.

Iceberg diagram: the surface a skim review sees versus the defects below the waterline

The data says the quality underneath is worse than it looks. Veracode’s 2025 GenAI Code Security Report tested 80 tasks across more than 100 models, and 45% of the generated solutions introduced an OWASP-class vulnerability. Cross-site scripting tasks failed 86% of the time, and newer, larger models did no better, so the next model release is unlikely to fix this. A Stanford study, Do Users Write More Insecure Code with AI Assistants?, found that participants using an AI assistant wrote less secure code and were more confident that it was secure.

On maintainability, GitClear’s analysis of 211 million changed lines from 2020 to 2024 shows copy-pasted code rising from 8.3% to 12.3% of changed lines, refactoring falling from about 25% to under 10%, and code rewritten within two weeks of landing nearly doubling. In 2024, copy-pasted lines outnumbered moved lines for the first time in their data. Our own impression of how things are going isn’t reliable either: in METR’s randomized trial, experienced developers were 19% slower with AI tools on their own repositories while believing they were 20% faster.

None of this means agents are a lost cause. I use them every day and won’t stop. It means the defects moved to places a quick look can’t find, at the same time the popular opinion says a quick look is enough.

The smells I look for in agent-written diffs

These patterns share one cause: the model has no sense of runtime cost. A line that adds a lot of runtime cost weighs the same to the model as a line that doesn’t. A repository call inside a loop reads as naturally as a batch fetch. Wrapping everything in try/catch looks careful.

SmellWhy the model produces itWhat it costs
N+1 queriesA fetch per item inside a loop reads naturally; batching needs a view of the whole functionThe agent doesn’t know how much data there is: one query per row is fast on a 3-row fixture and very slow on an account with 10k rows
Unoptimized queriesThe model writes a query that returns the right rows; it can’t see the table size, the indexes, or the query planFull table scans, SELECT * on wide tables, filtering and sorting in application code, no pagination; fast on local development data, very slow or timing out on millions of rows
Needless allocationChained map/filter/sorted is the most common pattern in training data, not the cheapestIntermediate lists, defensive copies, regexes compiled on every call; garbage collection runs more often and slows down the whole service
Needless loopingA contains or find inside a loop maps directly onto the spec’s wording; building a set or map first is a step nobody asked forO(n²) time, or O(n²) allocation when each pass copies the list; fine on the 10-item test fixture, minutes on 1 million real rows
Copy-paste abstractionsWriting a near-duplicate is easier for the model than finding and extending the existing helperslightly different versions of the same logic, these versions are a drift risk waiting to happen
try/catch everywhereCatch, log, and continue looks like production-grade error handlingCrashes turned into silent wrong answers; failures that appear far from their cause
Defensive dead codeUnused parameters “for future use”, null checks on values that can’t be null, comments that repeat the codeNoise that hides the one check that matters

None of these are visible at the intent level because the intent is satisfied. You find the N+1 in the diff, or you find it in an incident.

Intent review is necessary but not enough

I’m not arguing against intent review. It matters more now, not less. A common agent failure is building the wrong thing well: solving a nearby problem, quietly cutting scope, or “fixing” a failing test by deleting it or, worse, by mocking the response so it passes. Catching that needs what the spec-driven approach recommends: a written intent and a reviewer who checks the diff against it. The same goes for invariants, the rules every change must keep: this balance never goes negative, this handler is idempotent, these locks are always taken in this order. Agents break invariants easily because invariants live in people’s heads and in incident reports, not in the context window.

So review the intent and the invariants, then read the code anyway. Many different implementations satisfy the same intent, and the model picks one based on what looks likely, not on what is good. “Fetch the user’s transactions and render a statement” is satisfied by a version with one query and by a version with four hundred. It’s satisfied by a version that returns the error and by a version that swallows it and shows an empty statement, which looks exactly like a zero balance. Intent review tells you that you are building the right thing. Performance, failure behavior, security, and maintainability belong to the specific code the agent wrote, and only reading that code tells you about them.

Tests written by the same model that wrote the bug

“But the tests pass” is the strongest reply from the intent side, and it is weaker than it looks. When the model misunderstands the spec, it misunderstands it the same way in the code and in the tests. The implementation contains the mistake, and the test asserts it. A green suite then shows that the code agrees with itself, not that it is correct.

Research on self-review supports this. Self-Correction Bench found that models miss errors in their own output that they catch in other people’s, with an average blind-spot rate of 64.5% across fourteen models. A survey in TACL concluded that LLM self-correction without outside feedback mostly doesn’t work, and that many reported successes disappear under fair evaluation.

The tests are also often weaker than their coverage numbers suggest: only happy-path cases, so many mocks that no real code runs, assertions that only check that no exception was thrown. Researchers have shown suites with 100% line coverage and a 4% mutation score, meaning almost every bug they deliberately inserted passed the full suite. Mutation testing measures exactly this: it inserts small faults and counts how many the tests catch. Meta now uses LLMs to generate mutants at scale because coverage alone stopped telling them much. If an agent wrote your tests, the mutation score tells you whether they protect anything.

The correlated-failure loop: one model writes the code, the tests, and the review, so the blind spots align

Two diffs I keep rejecting

These are reconstructions written for this post, combining patterns I keep seeing in agent PRs. They aren’t copied from a real codebase, but they match diffs I have rejected.

The statement that queried the database 401 times. Intent: “render a monthly statement.” The agent’s version compiled, passed its own tests against a fixture with three transactions, and looked clean:

// Before: fluent, green, and O(n) database round-trips
fun monthlyStatement(accountId: Long): Statement {
    val account = accounts.findById(accountId).orElseThrow()
    val txns = transactions.findByAccountIdAndMonth(accountId, month)
    val lines = txns.map { txn ->
        val merchant = merchants.findById(txn.merchantId).orElseThrow() // N+1
        StatementLine(txn, merchant.displayName, merchant.category)
    }
    val sorted = lines.sortedByDescending { it.txn.timestamp }   // copy
    val nonZero = sorted.filter { it.txn.amount.signum() != 0 }  // copy
    return Statement(account, nonZero)
}

Nothing here is wrong against the intent. With a real account of 400 transactions, it runs one query per line plus two full copies of the list. The fix is the usual one:

// After: two queries, one pass
fun monthlyStatement(accountId: Long): Statement {
    val account = accounts.findById(accountId).orElseThrow()
    val txns = transactions.findByAccountIdAndMonth(accountId, month)
    val merchantsById = merchants.findAllById(txns.mapTo(HashSet()) { it.merchantId })
        .associateBy { it.id }
    val lines = txns.asSequence()
        .filter { it.amount.signum() != 0 }
        .sortedByDescending { it.timestamp }
        .map { StatementLine(it, merchantsById.getValue(it.merchantId)) }
        .toList()
    return Statement(account, lines)
}

Intent review can’t tell these two functions apart. A reviewer reading the code sees the repository call inside map in a few seconds.

The config loader that couldn’t fail. Intent: “load service config, with defaults.” The agent wrote this:

# Before: nothing can go wrong, which is the problem
def load_config(path):
    try:
        if path is not None and os.path.exists(path):
            try:
                with open(path) as f:
                    data = json.load(f)
            except (json.JSONDecodeError, OSError) as e:
                logger.error("Failed to parse config: %s", e)
                data = {}
        else:
            data = {}
    except Exception as e:
        logger.error("Unexpected error loading config: %s", e)
        data = {}
    return data or DEFAULT_CONFIG

It looks responsible, but it does the wrong thing. A corrupted config file (a bad deployment, a partial write) now starts the service with default settings and a log line nobody reads. In production, that is a service running with the wrong limits while its health checks pass. The correct version is shorter, and a bad file stops the boot:

# After: a corrupt config is a failed deploy, not a log line
def load_config(path: Path) -> dict:
    if not path.exists():
        return DEFAULT_CONFIG
    return json.loads(path.read_text())  # let it crash the boot

Both versions satisfy “load config with defaults.” Only reading the lines shows that the first one made a big design decision (failure should never be visible) and hid it inside the boilerplate.

What review has to cover now

Agent-era review doesn’t replace the lines with the intent. It covers more layers than before, which means review got more expensive while code got cheaper. Pretending otherwise moves the cost to whoever is on call.

The widened review funnel: intent, invariants, tests, lines, and what each layer catches

This doesn’t mean reading 900-line diffs line by line. It means refusing to accept 900-line diffs. Small changes with one purpose were always good practice; with agents, they are the only way to keep line review affordable. If review is your bottleneck, make the units of review smaller instead of reviewing less.

Where tooling helps

The line level is also the easiest layer to automate, as long as the tools don’t share the model’s blind spots.

Deterministic analyzers come first. Linters, type checkers, and static analysis (detekt, ruff, Semgrep, SonarQube) check rules, not plausibility. They catch part of the list above (duplicate blocks, broad exception catches, unused parameters) without any human time. When agents write the code, a strict analyzer config is the cheapest defense you have.

Measure runtime instead of guessing it. N+1s and extra allocations are hard to see in a diff and easy to see in instrumentation: query-count assertions in tests (assertNumQueries, Hibernate statistics), benchmark gates on hot paths, and profiling for anything that combines a loop with a datastore. A CI check that reports “this endpoint went from 3 queries to 403” settles the discussion.

Run mutation testing on agent-written suites. Pitest and mutmut tell you how many inserted bugs the suite actually catches, which coverage can’t.

Use a second agent as a reviewer and know its limits. A second model with fresh context and an adversarial prompt does catch real defects; I run one on every PR. Its usefulness depends on how independent it is, and it is only partly independent: similar training data, the same preference for code that looks right. As Aviator puts it, AI reviewing AI tends toward circular validation: the code looks correct, the tests pass, the review approves, and all three judgments came from the same source. Treat the agent reviewer as a fast filter before human review, not a replacement for it.

None of these tools replace one accountable person reading the code that is about to run, with the system’s history in their head. In Stack Overflow’s 2025 survey, 84% of developers use or plan to use AI tools, and more of them distrust the accuracy of the output (46%) than trust it (33%). Both numbers are true at once, and review is the last point where a responsible person looks at the actual code.

Review is the first time anyone thinks about the code

When a person writes a function, they decide what happens when the query returns a thousand rows, what happens when the config file is corrupt, and which existing helper to call. Those decisions exist in their head before the code exists in the diff, so a reviewer who skims is checking work that someone already thought through. A skim was enough because review was the second look.

Agent code contains the same decisions, but nobody made them; the model picked whatever looked likely. Nobody chose one query per row in monthlyStatement, and nobody chose to boot the service on default settings when the config is corrupt. Those choices appear for the first time in the diff, and the reviewer is the first person who sees them. That is why line review got more expensive, and why skipping it doesn’t remove the work. The decisions still get noticed, only later, in production, by someone with less context and less time.

I still let agents write most of my code. I read every line they write because that reading is the only point where anyone decides whether those lines are right.

References