Reviewing AI-Generated Code

Active since 2024.

As more code is written by machines, review becomes the place where quality is decided. That puts a lot of weight on a human process that was designed under the assumption that another human wrote the code, and could be asked what they meant.

We study what actually changes in that process: behaviourally, not aspirationally.

Reviewers spend longer, but do not look harder

In a Wizard-of-Oz experiment (Khojah et al., 2026), 32 practitioners reviewed Python pull requests in which some segments were labelled as generated by an LLM tool, and some were not. The same code appeared labelled for some participants and unlabelled for others, so any difference is attributable to the label rather than to the code. We recorded gaze with an eye tracker and followed up with exit interviews.

Gaze paths for one participant reviewing an unlabelled file (top) and an LLM-labelled file (bottom). Purple marks time inside the reviewed code, orange the label itself.

Thoroughness, measured through saccade lengths, did not change. Time did: participants fixated noticeably longer on code labelled as LLM-generated. Around a third of them also shifted what they were looking for, checking logical correctness more carefully in LLM-labelled code, or using the prompt as a guide to what the code was supposed to do. Which criteria they shifted to depended on their prior trust in AI: participants who trusted it more looked at readability and consistency, those who trusted it less went hunting for logic errors.

The gap this exposes is between stated intent and actual behaviour. Reviewers described their approach as consistent, and believed it was, while their attention was being pulled around by a label they said they were ignoring.

Developers accept suggestions largely as-is

The other half of the picture is what happens before review. We analysed 169 GitHub commits in which a developer refactored their code based on a linked ChatGPT conversation (Schön et al., 2026), measuring how much of the suggestion actually survived into the committed code.

How similar the committed code is to the suggestion it came from, under four similarity measures. The mass near 1.0 is suggestions adopted essentially unchanged.

Developers mostly accept the suggestion and use it without modification, typically after a short interaction of one to four prompts. When they do change it, the changes are substantial rather than cosmetic, and fall into five patterns that depend on the refactoring activity, the prompt, and whether the response was valid at all, most commonly dropping the part that introduced an error or some unrequested new behaviour, and keeping the rest.

Taken together: suggestions arrive largely unexamined, and the review that follows spends more time without applying more scrutiny.

In our ongoing work in this theme, we are interested in what review tooling should surface when nobody wrote the code, and in whether the effects above hold when the unit of review is an entire agent-authored change rather than a labelled segment.

Contacts:

Ranim Khojah

Dr. Francisco Gomes de Oliveira Neto

Dr. Philipp Leitner