Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions

We develop a two-stage LLM-assisted framework to determine whether review comments are addressed.

Repeats the framework that B6 states as a contribution, without the accuracy that B6 gives.

Among the remaining repositories, usage was highly concentrated on the four most popular actions.

Repeats the concentration result that B2 states, without a number of its own; C1 is the next sentence and gives the numbers.

Source Features represent the most significant group, showing a clear difference in developers’ responses based on the comment origins (human vs. actions; file-level vs. hunk-level).

Repeats the group-importance result that B5 states and C8 gives the ranking for, without a number of its own.

The figure for the human versus action origin, Is Human with directionality 0.99, stands only in Table XI (page 9) and in no sentence of the text.

In general, human-authored review comments are much more likely to be addressed than AI-generated ones: 60% of valid human review comments led to code changes, compared to only 0.9%–19.2% for valid AI-generated comments depending on the tool (Table IX).

Repeats the result that AI-generated comments lag behind human ones, which B4 states and C4 gives the numbers for.

For the least experienced developers, the addressing rate was five times higher than for the most experienced (Table XVI).

Repeats the contributor-experience result that B8 states, giving the comparison of the extreme bins as a ratio; R41 gives the same comparison as rates.

Even the best-performing AI tool achieved only a 19.2% addressing rate, far below the human reviewers’ 60%.

Repeats the result that AI-generated comments lag behind human ones, which B4 states and C4 gives the numbers for, using the top tool's rate.

Many AI-generated comments are not addressed, especially when they are vague or lack context.

Repeats the result that AI-generated comments largely go unaddressed, which B4 states with numbers.

The most frequently configured options were Prompt Context Augmentation (103/147, 70.1%), primarily for specifying the natural language of review comments, LLM Selection (95, 64.6%), and Input Settings (70, 47.6%).

B1 would still stand, because it says that 82.6% customized at least one optional parameter, and a different ranking of which options they chose leaves that share unchanged.

On average, these 178 repositories made changes to the configuration file 4 times after the initial setup.

B1 would still stand, because it is about customizing at least one parameter, not about how often the configuration file changed afterwards.

Of these, 68 repositories (38.2%) retained their original configurations without further changes.

B1 would still stand, because a repository that never changed its configuration again may still have customized a parameter at setup, which is what B1 counts.

Temporal analysis revealed that 50 projects (28.1%) finalized modifications within one week, and 20 (11.2%) within one month, while 40 repositories (22.5%) continued adjustments beyond one month.

B1 would still stand, because it says nothing about when repositories stopped adjusting their configurations.

The most frequently modified parameters were LLM Selection (25, 22.7% of 110 repositories that made post-setup changes), Prompt Customization (21, 19.1%), and LLM Hyperparameters (13, 11.8%).

B1 would still stand, because a different ranking of the parameters changed after setup leaves the share that customized at least one parameter unchanged.

Beyond action-specific options, we also observe that 50 repositories (45.5%) adjusted their workflow-level triggers by modifying the on or if conditions.

B1 would still stand, because it counts repositories that customized an optional action parameter, not repositories that edited workflow-level triggers.

Additionally, 12 repositories (10.9%) updated the action reference, such as replacing @main with a specific release tag.

B1 would still stand, because updating which version of an action is referenced is not customizing an optional parameter.

The top performers for Stage-1 were gpt-4.1 (94.5% average overall accuracy), deepseek-v3 (94.0%), and claude-3-sonnet (93.2%).

B6 would still stand; these are per-model figures from the round that chose the setup, and C5 and C6 are the figures the later analysis rests on.

For Stage-2, deepseek-r1 led with 95.4%, followed by openai-o3-mini (92.4%) and o4-mini (91.3%).

B6 would still stand; these are per-model figures from the round that chose the setup, and C5 and C6 are the figures the later analysis rests on.

Table VII reports the performance of our optimal cross-combined setup: gpt-4.1 for Stage-1 and o3-mini for Stage-2, which achieved strong and balanced performances across three comment sources, with an average of 86.1% overall accuracy and 74.6% Macro-F1 score under the full 6-class scheme.

B6 would still stand; this is the figure for the full 6-class scheme, while C5 is the figure for the Valid, Addressed versus all others distinction that the later analysis rests on.

The setup this sentence names, gpt-4.1 for Stage-1 and o3-mini for Stage-2, is the one C5 and C6 report on.

The lower macro-F1 for human-authored comments (62.4%) reflects stronger class imbalance within that subset (see Table VI), which penalizes macro-averaged metrics.

B6 would still stand; this is the figure for one comment source within the annotated sample, where C5 gives the average across the three sources.

The model achieved an average of 88.6% overall accuracy (Macro-F1=0.846) across the five test folds, indicating that our engineered features are effective predictors of comment addressing.

It is the accuracy of a model that the paper only reads other findings from, namely the SHAP importances and directions that B5 rests on.

Specifically, coderabbitai/ai-pr-reviewer achieved the highest valid and addressed rate (19.2%).

B4 would still stand, because it gives the range 6.5% to 19.2% across hunk-level actions and this is the rate of the single tool at the top of it.

Finally, positive correlations for Repository Features demonstrated that comments were more likely to be addressed in large and active projects (Repo File Size, ρ = 0.70; Repo Issue Count, ρ = 0.64, ranked 13/36).

B5 would still stand, because it names Source Features and Comment Features as the groups comment addressing is associated with, not Repository Features.

Interestingly, the LLM choice did not show high predictive importance (LLM GPT-3.5 ranked 19/36 and LLM GPT-4 ranked 21/36 in the feature list of the overall model; Is Model GPT4 ranked 23/33 in the feature list of the AI-specific model), indicating that simply employing a more advanced model does not decisively guarantee comment addressing.

B5 would still stand, because the Source Features it names are human versus AI and the specific action type, not which LLM the action used.

However, when comparing addressing rates between two model families (Table XIV), GPT-4 generated comments did show better addressing rates.

B5 would still stand, because it names human versus AI and the specific action type as its Source Features, not the LLM family.

Table XIV (page 10) prints, for the total, 5.4% of 1,435 GPT-3.5 comments and 12.6% of 1,555 GPT-4 comments addressed, with p <= 0.05, while R37 on the same page reports that the LLM-choice features ranked low in predictive importance.

Regarding topics, comments were addressed more often when they pertained to “Concurrency Control and Locking Robustness” (e.g., “using a ‘set-if-not-exists’ pattern”) (LDA Topic 2, ρ = 0.50), as well as “UI Usability and Accessibility Enhancements” (e.g., “adding aria-label for screen reader support”) (LDA Topic 1, ρ = 0.41).

B5 would still stand, because the Comment Features it names are conciseness and code-richness, not the topic a comment belongs to.

As shown in Table XV, the addressing rate for both human and AI-generated comments increased noticeably when Code Text Ratio exceeded 0.5.

B5 rests on the code-richness result over the whole sample, which C11 gives; this sentence gives the pattern for binned categories of Code Text Ratio.

Table XV (page 10) prints addressing rates across the five bins of 4.2%, 7.2%, 5.7%, 10.5%, and 23.2% for the four actions and 71.2%, 65.6%, 75.8%, 78.5%, and 89.9% for human comments.

As shown in Table XVI, valid comments directed at project newcomers (Author Prior Commits ≤ 124) achieved 16% addressing rate, compared to just 3.3% for the most experienced contributors.

B8 rests on the contributor-experience result over the whole sample, which C12 gives; this sentence gives the rates for two of the binned experience categories.

The text and Table XVI give the newcomer figure differently. This sentence (page 10) prints 16%, while Table XVI (page 10) prints 16.1% for the (-0.00, 30] bin and 16.1% for the (30, 124] bin; the table prints 3.3% for the (1013, 4,316] bin.

We found that in the two actions with manually triggered comments (Action ID-1 and ID-2), manually triggered comments consistently showed higher addressing rates.

B7 rests on the trigger-mode result over the whole sample, which C9 gives; this sentence gives the comparison for the two actions that have manually triggered comments.

The sentence gives no number of its own. Table XIII (page 10) holds them: 6.8% of 1,595 automatic versus 12.8% of 86 manual comments for ID-1, and 0.5% of 602 versus 22.2% of 18 for ID-2, both with p <= 0.05.

Notably, human reviews mainly focused on these newcomers, with 79% of comments directed at them, aligning with typical community practices.

B8 would still stand, because it is about which comments are addressed, not about which authors human reviewers chose to comment on.

Among the four actions we analyzed in depth, one stood out: coderabbitai/ai-pr-reviewer achieved a 19.2% rate of valid comments addressed, outperforming other tools by a large margin.

B4 would still stand, because it gives the range across hunk-level actions and this is the rate of the single tool at the top of that range.

For example, mattzcarey/code-review-gpt had only 0.9% of valid comments addressed.

B4 would still stand, because it gives the range across file-level actions and this is the rate of the single tool at the bottom of that range.

Only 1 comment was partially addressed, which provided a specific performance improvement: “For better performance, preallocate the ‘architectures’ slice by using ‘make([]types.Architecture, 0, len(fm.Spec.Architectures))’ ”.

B4 would still stand, because it compares hunk-level with file-level actions and does not mention PR-level actions, which this sample is drawn from.

Section V-A (page 11) uses this sample to support its conclusion that fine-grained, hunk-level reviews are more effective, so the checker can turn it into a claim if that is taken as a basis of B4.

For example, anc95/ChatGPT-CodeReview showed a 12.8% addressing rate for manually triggered comments versus 6.8% for automatically triggered ones (Table XIII).

B7 would still stand, because C9 gives the trigger-mode correlation over the whole sample and this is the comparison for a single action.

To assess representativeness, the first author examined an additional 250 samples and observed similar addressing patterns as reported in our main findings.

B4 would still stand, because this is a robustness check that reports no change in the addressing patterns.

We present a large-scale empirical study of 16 popular AI-based code-review actions for GitHub workflows, analyzing more than 22,000 review comments in 178 repositories.

Every number in this summary sentence describes the study: the actions studied, the comments analyzed, and the repositories they come from.

From a review of approximately 240 actions, we identified an initial set of 20 candidates related to AI-driven code review.

Describes the study: how many Marketplace actions were screened and how many were shortlisted.

Out of 718 matched repositories, 178 met the maturity criterion (≥ 50 PRs), and these contained a total of 22,326 AI-generated review comments.

Describes the study: the repositories matched and retained and the comments they contain, in more detail than the summary sentences give.

We found that 37.1% of the mature repositories declared an action but showed no generated comments, indicating a gap between declaration and actual use.

No broad statement states a main result about the gap between declaring an action and generating comments.

B1 states the configuration result and B2 the concentration result, and neither mentions this gap. Section IV-A (page 4) builds on it only to introduce the concentration sentence that C1 gives the numbers for, so the checker can turn it into a claim if the gap is taken as a main result.

We analyzed 178 mature repositories and found a total of 22,326 AI-generated review comments.

Both numbers in this boxed sentence describe the study: the repositories analyzed and the comments collected from them.

After these filters, only Actions ID-1 to ID-4 retained sufficient data for the addressing analysis, with 2,908, 399, 713, and 169 comments respectively.

Describes the study: how much material survived the filters for the addressing analysis.

The text and Table V give the count for Action ID-2 differently. This sentence (page 5) prints 399 for ID-2, while Table V (page 5) prints Total Comments 773 for mattzcarey/code-review-gpt.

In total, our comment addressing dataset includes 5,652 review comments: 3,604 from file-level review actions, 882 from hunk-level actions, and 1,166 authored by human reviewers.

Describes the study: the size of the dataset the RQ2 results rest on, split into its three kinds of comment.

Specifically, we randomly sampled 50 comments from each category where subsequent file modifications occurred (i.e. 467 eligible from file-level actions, 424 eligible from hunk-level actions, and 1,070 eligible human-authored comments), yielding 150 comments in total.

Describes the study: the sample that was annotated by hand and the pool it was drawn from.

Inter-rater agreement (Cohen’s κ) reached substantial levels over the full 6-class annotation scheme (i.e., “None”, “General”, “Valid-Uncertain”, “Valid-Unaddressed”, “Valid-Partially”, “Valid-Fully”): 0.674 for file-level actions, 0.734 for hunk-level actions, and 0.764 for human reviews.

Describes the study: an agreement score that supports the annotation procedure behind the ground truth.

Furthermore, when analyzing the complete dataset, we found that a significant portion of comments (87.0% of file-level, 51.9% of hunk-level, and 8.2% of human comments) had no subsequent modifications.

Describes the study: the share of the dataset in which the reviewed file was never changed afterwards.

In contrast, the automated annotation was applied to the entire comment dataset, which included a large volume of comments (65.3%) with no subsequent file changes.

Describes the study: the share of the labeled dataset with no subsequent file change, given here for the dataset as a whole.

Following the best-performing setup on the annotated dataset, we used gpt-4.1 to assess the validity of all 5,652 comments, identifying 3,955 as valid, and then used openai-o3-mini to assess whether these valid comments had been addressed.

Describes the study: how much material the framework labeled and how much of it it judged valid.

We focus specifically on the 3,879 valid comments from RQ2 that have a definitive addressing label (excluding 76 uncertain cases), examining the factors that determine whether they are addressed.

Describes the study: the comments the factor analysis rests on and the ones excluded from it.

After checking for multicollinearity, we removed 9 redundant features, yielding a final set of 36 features presented in Table X.

Describes the study: the size of the feature set the factor analysis uses.

To enable focused analysis, we mapped the 3,879 comments into a binary classification task: comments labeled as “Valid-Partially” or “Valid-Fully” were grouped as the “Addressed” class (25.1%), with “Valid-Unaddressed” comments forming the “Not Addressed” class (74.9%).

Describes the study: how the labeled comments were split into the two classes the classifier was trained on.

The resulting AI-Specific Model achieved an overall accuracy of 91.7% and a Macro-F1 score of 0.647 in 5-fold cross-validation.

It is the accuracy of a model that the paper only reads other findings from, namely the SHAP results for AI-generated comments.

To further investigate the effectiveness of review granularity, we randomly investigated 14.4%, i.e., 30 comments, of the 208 PR-level review comments as follows: 5 from Action ID-8, 5 from ID-11, and 20 from ID-16.

Describes the study: the sample of PR-level comments inspected by hand and the pool it came from.

We observed a significantly lower addressing rate for AI comments in closed PRs compared to merged ones: only 3.0% (17/570) for Action ID-1, and 0% for Action ID-2 (n = 14), ID-3 (n = 18), and ID-4 (n = 0).

No broad statement states a main result about closed pull requests.

Section V-B (page 12) uses this supplementary analysis to justify focusing the main results on merged pull requests.

We identified a total of 50 replies and manually categorized them into four types:

Describes the study: how much material the supplementary reply analysis coded.

Specifically, Willingness to Follow (10%) and Prompting Reflection (14%) imply that even if not addressed immediately, a comment may still trigger code changes in the future.

No broad statement states a main result about impact beyond immediate code changes.

The conclusion (page 15) states that AI comments can still prompt developer reflection and inform future improvements, but it gives no quantity, so it is recorded as no broad statement; the checker can turn this sentence into a claim if that conclusion sentence is taken as a main result.

We also tried logistic regression in our experiments and it achieved a lower performance (86.1% accuracy and 0.826 Macro-F1 in 5-fold cross-validation) than our Random Forest model (88.6% and 0.846 respectively).

These are the accuracies of models that the paper only reads other findings from, compared here to justify the choice of classifier.

As described in RQ2 Phase I, we began with 16,762 comments from 100 repositories. A language filter reduced this to 4,229 comments from 53 repositories, excluding about 75% of comments and 47 repositories.

Describes the study: the material before and after the language filter.

Two passages give the count before the language filter differently. This sentence (page 14) prints 16,762 comments, while Section IV-B Phase I (page 5) prints that retaining merged-PR comments yielded 16,816 comments and that the language filter removed 12,587 of them.