# Claims: Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions Paper `tse26-ai-code-review` (17 pages, `evals/papers/tse26-ai-code-review.pdf`): 8 broad statements, 12 narrow claims, 55 rejected candidates. Generated from `claims.json` by `cea_claims.py render`. To change the record, edit `claims.json` and render again. ## Broad statements ### B1: rq_answer, page 5 > Among the 178 repositories, 82.6% customized at least one optional parameter, showing that developers often go beyond default settings when using these actions. - Section: IV-A RQ1, Answer to RQ1 - Narrow claims: C2 ### B2: conclusion, page 15 > Our findings reveal that adoption is highly concentrated on four popular actions. - Section: VIII Conclusion and Future Work - Narrow claims: C1 ### B3: abstract, page 1 > We found that while adoption is growing, its effectiveness varies widely. - Section: Abstract - Narrow claims: none selected - States: We found that adoption is growing - Note: The sentence states two main results, and B4 records the second, that effectiveness varies. B3 records the first, that adoption is growing. No claim serves it: the paper measures adoption at one point in time, for each action, across repositories, pull requests, and comments (Table IV, page 4), and reports no measurement of adoption over time. ### B4: rq_answer, page 8 > We found that hunk-level review actions (6.5%–19.2%) exhibit a higher addressing rate compared to file-level actions (0.9%–4.2%), yet the addressing rate of AI-generated review comments still lags behind human review comments (60%). - Section: IV-B RQ2, Answer to RQ2 - Narrow claims: C3, C4 ### B5: rq_answer, page 10 > Comment addressing is strongly associated with Source Features (human vs. AI and specific action type) and Comment Features (conciseness and code-richness are better), highlighting the importance of thoughtful automated review design. - Section: IV-C RQ3, Answer to RQ3 - Narrow claims: C7, C8, C10, C11 ### B6: contributions, page 2 > We introduce an LLM-assisted framework for assessing whether code review comments are actionable and whether they have been addressed, achieving high accuracy compared to human annotations. - Section: I Introduction, contribution list - Narrow claims: C5, C6 ### B7: abstract, page 1 > Comments that are concise, contain code snippets, and are manually triggered, particularly those from hunk-level review tools, are more likely to result in code changes. - Section: Abstract - Narrow claims: C9 - States: Comments that are manually triggered are more likely to result in code changes. - Note: B5 states the conciseness and code-snippet part of this sentence and B4 the hunk-level part. Manual triggering is named by no other broad statement, so B7 is recorded for that clause. Section V-A (page 11) lists manual triggering as one of the design characteristics the study identifies. ### B8: conclusion, page 15 > However, comments that are concise, specific are more likely to be addressed—particularly by less experienced contributors. - Section: VIII Conclusion and Future Work - Narrow claims: C12 - States: comments are more likely to be addressed—particularly by less experienced contributors - Note: B5 states the conciseness and specificity part of this sentence. The contributor-experience part is named by no other broad statement, so B8 is recorded for it. Section V-A (page 11) lists it as one of the design characteristics, under the heading Focus on newcomers. ## Narrow claims Grouped under each main result they serve, the result the paper states in the most places first. Within a result the claims stand in page order, because the paper puts no order on them. ### B4 (rq_answer, stated in 4 places): We found that hunk-level review actions (6.5%–19.2%) exhibit a higher addressing rate compared to file-level actions (0.9%–4.2%), yet the addressing rate of AI-generated review comments still lags behind human review comments (60%). - C3, page 8, 1 of 2 for this result - C4, page 8, 1 of 2 for this result ### B5 (rq_answer, stated in 2 places): Comment addressing is strongly associated with Source Features (human vs. AI and specific action type) and Comment Features (conciseness and code-richness are better), highlighting the importance of thoughtful automated review design. - C8, page 9, 1 of 4 for this result - C7, page 10, 1 of 4 for this result - C10, page 10, 1 of 4 for this result - C11, page 10, 1 of 4 for this result ### B6 (contributions, stated in 2 places): We introduce an LLM-assisted framework for assessing whether code review comments are actionable and whether they have been addressed, achieving high accuracy compared to human annotations. - C5, page 7, 1 of 2 for this result - C6, page 8, 1 of 2 for this result ### B2 (conclusion, stated in 2 places): Our findings reveal that adoption is highly concentrated on four popular actions. - C1, page 4-5, the only claim for this result ### B8 (conclusion, stated in 2 places): However, comments that are concise, specific are more likely to be addressed—particularly by less experienced contributors. - C12, page 10, the only claim for this result ### B1 (rq_answer, stated in 1 place): Among the 178 repositories, 82.6% customized at least one optional parameter, showing that developers often go beyond default settings when using these actions. - C2, page 5, the only claim for this result ### B7 (abstract, stated in 1 place): Comments that are concise, contain code snippets, and are manually triggered, particularly those from hunk-level review tools, are more likely to result in code changes. - C9, page 10, the only claim for this result ### Every claim #### C1: page 4-5, IV-A RQ1, Results > Actions ID-1 to ID-4 (including two file-level and two hunk-level [...] reviewers) accounted for 91.1% of the reviewed repositories, 95.2% of the pull requests, and 98.9% of the generated comments. - Serves: B2 - Selection reason: B2 says adoption is highly concentrated on four popular actions. If these four actions had not accounted for the large majority of reviewed repositories, pull requests, and comments, B2 would fail. - Note: The sentence runs across the page break from page 4 to page 5 with Table V printed between its lines, which is what the [...] marks. Table IV (page 4) prints the per-action counts of repositories, pull requests, and comments, but not these three percentages. #### C2: page 5, IV-A RQ1, Results > Regarding configuration evolution, the majority of the mature repositories (147/178, 82.6%) customized at least one optional parameter. - Serves: B1 - Selection reason: B1 says 82.6% of the 178 repositories customized at least one optional parameter, showing that developers go beyond default settings. If that share were not a majority, B1 would fail. #### C5: page 7, IV-B RQ2, Results > Importantly, performance on the primary downstream distinction (“Valid, Addressed” vs. all others) remains high, with an average of 91.5% overall accuracy and 89.8% macro-F1. - Serves: B6 - Selection reason: B6 presents the LLM-assisted framework as a contribution and says it achieves high accuracy compared to human annotations. This is the figure for the Valid, Addressed versus all others distinction that the RQ3 analysis rests on. If it were not high, B6 would fail. - Note: This figure is measured on the 150 manually annotated comments and averaged across the three comment sources; Table VII (page 7) prints 92.0%, 91.6%, and 90.8% for the three sources and 91.5% as the average. C6 is the figure for the same distinction measured on the whole dataset. #### C3: page 8, IV-B RQ2, Results > Overall, hunk-level review actions exhibit a higher addressing rate (6.5%–19.2%) compared to file-level actions (0.9%–4.2%). - Serves: B4 - Selection reason: B4 says hunk-level review actions have a higher addressing rate than file-level actions. If that ordering reversed, B4 would fail. - Note: Table IX (page 7) prints the per-action shares of Valid, Addressed comments that these ranges cover: 4.2% for ID-1, 0.9% for ID-2, 19.2% for ID-3, and 6.5% for ID-4. #### C4: page 8, IV-B RQ2, Results > Beyond this discrepancy, both tables indicate that the addressing rate of AI-generated review comments (0.9%–19.2% in Table IX) still lags behind human review comments (60%). - Serves: B4 - Selection reason: B4 says the addressing rate of AI-generated comments lags behind human review comments. If the AI rates had not been below the human rate, B4 would fail. - Note: Table IX (page 7) prints 60.0% as the Valid, Addressed share for the 1,166 human comments. #### C6: page 8, IV-B RQ2, Results > Considering the natural class imbalance in real-world review data, the overall classification accuracy across the full dataset is 97%, ensuring the reliability of our subsequent large-scale analysis in Phase IV (detailed breakdown in Table VIII). - Serves: B6 - Selection reason: B6 presents the LLM-assisted framework as a contribution and says it achieves high accuracy compared to human annotations. This is the figure for the same Valid, Addressed versus all others distinction measured on the full dataset that Phase IV labels. If it were not high, B6 would fail. - Note: This figure is measured on all 5,652 comments. Table VIII (page 7) prints Total Overall 97.0%, made up of 100% for the 3,691 comments with no code change and 91.3% for the 1,961 with code change. C5 is the figure for the same distinction measured on the annotated sample. #### C8: page 9, IV-C RQ3, Results > Notably, “Source Features” and “CommentP Features” rank first and second in total importance ( |ϕ|), suggesting that both who provides the feedback (source credibility) and what the feedback contains (content quality) are strongly associated with whether comments are addressed. - Serves: B5 - Selection reason: B5 says comment addressing is strongly associated with Source Features and Comment Features. If these two groups had not ranked first and second in total SHAP importance, B5 would fail. - Note: The extraction garbled the sentence: the summation sign of the total SHAP importance came out inside the word as “CommentP Features” and again as the space before |ϕ|. Table XI (page 9) prints the group totals 0.1589 for Source Features and 0.0959 for Comment Features. #### C7: page 10, IV-C RQ3, Results > The negative correlation for Is File Level Action (ρ = −0.95) aligns with the trend in Table IX, where hunk-level review actions generally outperform file-level counterparts. - Serves: B5 - Selection reason: B5 rests on the Source Feature results together, and this is the one for the specific action type that B5 names. If the correlation for Is File Level Action were not negative, that basis of B5 would disappear. - Note: Table XI (page 9) prints Is File Level Action at rank 3 with importance 0.0346 and directionality -0.95. #### C9: page 10, IV-C RQ3, Results > Automatic triggers also show a negative correlation with addressing (Trigger auto, ρ = −0.96). - Serves: B7 - Selection reason: B7 says manually triggered comments are more likely to result in code changes. If the correlation for automatic triggering were not negative, B7 would fail. - Note: Table XI (page 9) prints Trigger auto at rank 2 with importance 0.0410 and directionality -0.96. Table XIII (page 10) gives the addressing rates by trigger mode: 9.0% of 2,886 automatically triggered comments and 14.4% of 104 manually triggered ones, with p > 0.05 for the total and p <= 0.05 for ID-1 and ID-2. #### C10: page 10, IV-C RQ3, Results > For Comment Features, we find the comments were more likely to be addressed if they were concise (Text Length, ρ = −0.24) or contained a higher proportion of code (Code Text Ratio, ρ = 0.89), especially when they included multi-line code blocks(Has Multiline Code, ρ = 0.67 not shown in the table). - Part: For Comment Features, we find the comments were more likely to be addressed if they were concise (Text Length, ρ = −0.24). - Split from S1, with C11 - Serves: B5 - Selection reason: B5 names conciseness as one of the Comment Features that comment addressing is associated with. If the correlation for Text Length were not negative, that part of B5 would fail. - Note: Table XI (page 9) prints Text Length at rank 7 with importance 0.0133 and directionality -0.24, which is below the |ρ| > 0.3 threshold the paper states on page 9 for a meaningful correlation. #### C11: page 10, IV-C RQ3, Results > For Comment Features, we find the comments were more likely to be addressed if they were concise (Text Length, ρ = −0.24) or contained a higher proportion of code (Code Text Ratio, ρ = 0.89), especially when they included multi-line code blocks(Has Multiline Code, ρ = 0.67 not shown in the table). - Part: For Comment Features, we find the comments were more likely to be addressed if they contained a higher proportion of code (Code Text Ratio, ρ = 0.89), especially when they included multi-line code blocks(Has Multiline Code, ρ = 0.67 not shown in the table). - Split from S1, with C10 - Serves: B5 - Selection reason: B5 names code-richness as one of the Comment Features that comment addressing is associated with. If the correlation for Code Text Ratio were not positive, that part of B5 would fail. - Note: Table XI (page 9) prints Code Text Ratio at rank 6 with importance 0.0155 and directionality 0.89. The sentence itself says the Has Multiline Code figure is not in the table, and Table XI does not print it. #### C12: page 10, IV-C RQ3, Results > Within Modification Features, review comments targeting commits by experienced contributors were less likely to be addressed (Author Prior Commits, ρ = −0.69), suggesting that AI-generated reviews may be more helpful for newcomers. - Serves: B8 - Selection reason: B8 says comments are more likely to be addressed by less experienced contributors. If the correlation for Author Prior Commits were not negative, B8 would fail. - Note: Table XI (page 9) prints Author Prior Commits at rank 4 with importance 0.0235 and directionality -0.69. ## Rejected candidates ### R1: page 1, Abstract > We present a large-scale empirical study of 16 popular AI-based code-review actions for GitHub workflows, analyzing more than 22,000 review comments in 178 repositories. - Reason: Every number in this summary sentence describes the study: the actions studied, the comments analyzed, and the repositories they come from. ### R2: page 1, Abstract > We develop a two-stage LLM-assisted framework to determine whether review comments are addressed. - Repeats: B6 - Reason: Repeats the framework that B6 states as a contribution, without the accuracy that B6 gives. ### R4: page 3, III Study Setup, Action Selection > From a review of approximately 240 actions, we identified an initial set of 20 candidates related to AI-driven code review. - Reason: Describes the study: how many Marketplace actions were screened and how many were shortlisted. ### R5: page 4, IV-A RQ1, Results > Out of 718 matched repositories, 178 met the maturity criterion (≥ 50 PRs), and these contained a total of 22,326 AI-generated review comments. - Reason: Describes the study: the repositories matched and retained and the comments they contain, in more detail than the summary sentences give. ### R6: page 4, IV-A RQ1, Results > We found that 37.1% of the mature repositories declared an action but showed no generated comments, indicating a gap between declaration and actual use. - Reason: No broad statement states a main result about the gap between declaring an action and generating comments. - Note: B1 states the configuration result and B2 the concentration result, and neither mentions this gap. Section IV-A (page 4) builds on it only to introduce the concentration sentence that C1 gives the numbers for, so the checker can turn it into a claim if the gap is taken as a main result. ### R7: page 4, IV-A RQ1, Results > Among the remaining repositories, usage was highly concentrated on the four most popular actions. - Repeats: B2, C1 - Reason: Repeats the concentration result that B2 states, without a number of its own; C1 is the next sentence and gives the numbers. ### R3: page 5, IV-A RQ1, Answer to RQ1 > We analyzed 178 mature repositories and found a total of 22,326 AI-generated review comments. - Reason: Both numbers in this boxed sentence describe the study: the repositories analyzed and the comments collected from them. ### R8: page 5, IV-A RQ1, Results > The most frequently configured options were Prompt Context Augmentation (103/147, 70.1%), primarily for specifying the natural language of review comments, LLM Selection (95, 64.6%), and Input Settings (70, 47.6%). - Reason: B1 would still stand, because it says that 82.6% customized at least one optional parameter, and a different ranking of which options they chose leaves that share unchanged. ### R9: page 5, IV-A RQ1, Results > On average, these 178 repositories made changes to the configuration file 4 times after the initial setup. - Reason: B1 would still stand, because it is about customizing at least one parameter, not about how often the configuration file changed afterwards. ### R10: page 5, IV-A RQ1, Results > Of these, 68 repositories (38.2%) retained their original configurations without further changes. - Reason: B1 would still stand, because a repository that never changed its configuration again may still have customized a parameter at setup, which is what B1 counts. ### R11: page 5, IV-A RQ1, Results > Temporal analysis revealed that 50 projects (28.1%) finalized modifications within one week, and 20 (11.2%) within one month, while 40 repositories (22.5%) continued adjustments beyond one month. - Reason: B1 would still stand, because it says nothing about when repositories stopped adjusting their configurations. ### R12: page 5, IV-A RQ1, Results > The most frequently modified parameters were LLM Selection (25, 22.7% of 110 repositories that made post-setup changes), Prompt Customization (21, 19.1%), and LLM Hyperparameters (13, 11.8%). - Reason: B1 would still stand, because a different ranking of the parameters changed after setup leaves the share that customized at least one parameter unchanged. ### R13: page 5, IV-A RQ1, Results > Beyond action-specific options, we also observe that 50 repositories (45.5%) adjusted their workflow-level triggers by modifying the on or if conditions. - Reason: B1 would still stand, because it counts repositories that customized an optional action parameter, not repositories that edited workflow-level triggers. ### R14: page 5, IV-A RQ1, Results > Additionally, 12 repositories (10.9%) updated the action reference, such as replacing @main with a specific release tag. - Reason: B1 would still stand, because updating which version of an action is referenced is not customizing an optional parameter. ### R15: page 5, IV-B RQ2, Phase I > After these filters, only Actions ID-1 to ID-4 retained sufficient data for the addressing analysis, with 2,908, 399, 713, and 169 comments respectively. - Reason: Describes the study: how much material survived the filters for the addressing analysis. - Note: The text and Table V give the count for Action ID-2 differently. This sentence (page 5) prints 399 for ID-2, while Table V (page 5) prints Total Comments 773 for mattzcarey/code-review-gpt. ### R16: page 6, IV-B RQ2, Phase I > In total, our comment addressing dataset includes 5,652 review comments: 3,604 from file-level review actions, 882 from hunk-level actions, and 1,166 authored by human reviewers. - Reason: Describes the study: the size of the dataset the RQ2 results rest on, split into its three kinds of comment. ### R17: page 6, IV-B RQ2, Phase II > Specifically, we randomly sampled 50 comments from each category where subsequent file modifications occurred (i.e. 467 eligible from file-level actions, 424 eligible from hunk-level actions, and 1,070 eligible human-authored comments), yielding 150 comments in total. - Reason: Describes the study: the sample that was annotated by hand and the pool it was drawn from. ### R18: page 6, IV-B RQ2, Phase II > Inter-rater agreement (Cohen’s κ) reached substantial levels over the full 6-class annotation scheme (i.e., “None”, “General”, “Valid-Uncertain”, “Valid-Unaddressed”, “Valid-Partially”, “Valid-Fully”): 0.674 for file-level actions, 0.734 for hunk-level actions, and 0.764 for human reviews. - Reason: Describes the study: an agreement score that supports the annotation procedure behind the ground truth. ### R19: page 7, IV-B RQ2, Results > The top performers for Stage-1 were gpt-4.1 (94.5% average overall accuracy), deepseek-v3 (94.0%), and claude-3-sonnet (93.2%). - Reason: B6 would still stand; these are per-model figures from the round that chose the setup, and C5 and C6 are the figures the later analysis rests on. ### R20: page 7, IV-B RQ2, Results > For Stage-2, deepseek-r1 led with 95.4%, followed by openai-o3-mini (92.4%) and o4-mini (91.3%). - Reason: B6 would still stand; these are per-model figures from the round that chose the setup, and C5 and C6 are the figures the later analysis rests on. ### R21: page 7, IV-B RQ2, Results > Table VII reports the performance of our optimal cross-combined setup: gpt-4.1 for Stage-1 and o3-mini for Stage-2, which achieved strong and balanced performances across three comment sources, with an average of 86.1% overall accuracy and 74.6% Macro-F1 score under the full 6-class scheme. - Reason: B6 would still stand; this is the figure for the full 6-class scheme, while C5 is the figure for the Valid, Addressed versus all others distinction that the later analysis rests on. - Note: The setup this sentence names, gpt-4.1 for Stage-1 and o3-mini for Stage-2, is the one C5 and C6 report on. ### R22: page 7, IV-B RQ2, Results > The lower macro-F1 for human-authored comments (62.4%) reflects stronger class imbalance within that subset (see Table VI), which penalizes macro-averaged metrics. - Reason: B6 would still stand; this is the figure for one comment source within the annotated sample, where C5 gives the average across the three sources. ### R23: page 7-8, IV-B RQ2, Results > Furthermore, when analyzing the complete dataset, we found that a significant portion of comments (87.0% of file-level, 51.9% of hunk-level, and 8.2% of human comments) had no subsequent modifications. - Reason: Describes the study: the share of the dataset in which the reviewed file was never changed afterwards. ### R24: page 8, IV-B RQ2, Results > In contrast, the automated annotation was applied to the entire comment dataset, which included a large volume of comments (65.3%) with no subsequent file changes. - Reason: Describes the study: the share of the labeled dataset with no subsequent file change, given here for the dataset as a whole. ### R25: page 8, IV-B RQ2, Results > Following the best-performing setup on the annotated dataset, we used gpt-4.1 to assess the validity of all 5,652 comments, identifying 3,955 as valid, and then used openai-o3-mini to assess whether these valid comments had been addressed. - Reason: Describes the study: how much material the framework labeled and how much of it it judged valid. ### R46: page 8, IV-C RQ3, Approach > We focus specifically on the 3,879 valid comments from RQ2 that have a definitive addressing label (excluding 76 uncertain cases), examining the factors that determine whether they are addressed. - Reason: Describes the study: the comments the factor analysis rests on and the ones excluded from it. ### R47: page 8, IV-C RQ3, Phase I > After checking for multicollinearity, we removed 9 redundant features, yielding a final set of 36 features presented in Table X. - Reason: Describes the study: the size of the feature set the factor analysis uses. ### R48: page 9, IV-C RQ3, Phase II > To enable focused analysis, we mapped the 3,879 comments into a binary classification task: comments labeled as “Valid-Partially” or “Valid-Fully” were grouped as the “Addressed” class (25.1%), with “Valid-Unaddressed” comments forming the “Not Addressed” class (74.9%). - Reason: Describes the study: how the labeled comments were split into the two classes the classifier was trained on. ### R49: page 9, IV-C RQ3, Phase II > The model achieved an average of 88.6% overall accuracy (Macro-F1=0.846) across the five test folds, indicating that our engineered features are effective predictors of comment addressing. - Reason: It is the accuracy of a model that the paper only reads other findings from, namely the SHAP importances and directions that B5 rests on. ### R50: page 9, IV-C RQ3, Phase III > The resulting AI-Specific Model achieved an overall accuracy of 91.7% and a Macro-F1 score of 0.647 in 5-fold cross-validation. - Reason: It is the accuracy of a model that the paper only reads other findings from, namely the SHAP results for AI-generated comments. ### R29: page 10, IV-C RQ3, Results > Specifically, coderabbitai/ai-pr-reviewer achieved the highest valid and addressed rate (19.2%). - Reason: B4 would still stand, because it gives the range 6.5% to 19.2% across hunk-level actions and this is the rate of the single tool at the top of it. ### R36: page 10, IV-C RQ3, Results > Finally, positive correlations for Repository Features demonstrated that comments were more likely to be addressed in large and active projects (Repo File Size, ρ = 0.70; Repo Issue Count, ρ = 0.64, ranked 13/36). - Reason: B5 would still stand, because it names Source Features and Comment Features as the groups comment addressing is associated with, not Repository Features. ### R37: page 10, IV-C RQ3, Results > Interestingly, the LLM choice did not show high predictive importance (LLM GPT-3.5 ranked 19/36 and LLM GPT-4 ranked 21/36 in the feature list of the overall model; Is Model GPT4 ranked 23/33 in the feature list of the AI-specific model), indicating that simply employing a more advanced model does not decisively guarantee comment addressing. - Reason: B5 would still stand, because the Source Features it names are human versus AI and the specific action type, not which LLM the action used. ### R38: page 10, IV-C RQ3, Results > However, when comparing addressing rates between two model families (Table XIV), GPT-4 generated comments did show better addressing rates. - Reason: B5 would still stand, because it names human versus AI and the specific action type as its Source Features, not the LLM family. - Note: Table XIV (page 10) prints, for the total, 5.4% of 1,435 GPT-3.5 comments and 12.6% of 1,555 GPT-4 comments addressed, with p <= 0.05, while R37 on the same page reports that the LLM-choice features ranked low in predictive importance. ### R39: page 10, IV-C RQ3, Results > Regarding topics, comments were addressed more often when they pertained to “Concurrency Control and Locking Robustness” (e.g., “using a ‘set-if-not-exists’ pattern”) (LDA Topic 2, ρ = 0.50), as well as “UI Usability and Accessibility Enhancements” (e.g., “adding aria-label for screen reader support”) (LDA Topic 1, ρ = 0.41). - Reason: B5 would still stand, because the Comment Features it names are conciseness and code-richness, not the topic a comment belongs to. ### R40: page 10, IV-C RQ3, Results > As shown in Table XV, the addressing rate for both human and AI-generated comments increased noticeably when Code Text Ratio exceeded 0.5. - Reason: B5 rests on the code-richness result over the whole sample, which C11 gives; this sentence gives the pattern for binned categories of Code Text Ratio. - Note: Table XV (page 10) prints addressing rates across the five bins of 4.2%, 7.2%, 5.7%, 10.5%, and 23.2% for the four actions and 71.2%, 65.6%, 75.8%, 78.5%, and 89.9% for human comments. ### R41: page 10, IV-C RQ3, Results > As shown in Table XVI, valid comments directed at project newcomers (Author Prior Commits ≤ 124) achieved 16% addressing rate, compared to just 3.3% for the most experienced contributors. - Reason: B8 rests on the contributor-experience result over the whole sample, which C12 gives; this sentence gives the rates for two of the binned experience categories. - Note: The text and Table XVI give the newcomer figure differently. This sentence (page 10) prints 16%, while Table XVI (page 10) prints 16.1% for the (-0.00, 30] bin and 16.1% for the (30, 124] bin; the table prints 3.3% for the (1013, 4,316] bin. ### R44: page 10, IV-C RQ3, Results > We found that in the two actions with manually triggered comments (Action ID-1 and ID-2), manually triggered comments consistently showed higher addressing rates. - Reason: B7 rests on the trigger-mode result over the whole sample, which C9 gives; this sentence gives the comparison for the two actions that have manually triggered comments. - Note: The sentence gives no number of its own. Table XIII (page 10) holds them: 6.8% of 1,595 automatic versus 12.8% of 86 manual comments for ID-1, and 0.5% of 602 versus 22.2% of 18 for ID-2, both with p <= 0.05. ### R45: page 10, IV-C RQ3, Results > Source Features represent the most significant group, showing a clear difference in developers’ responses based on the comment origins (human vs. actions; file-level vs. hunk-level). - Repeats: B5, C8 - Reason: Repeats the group-importance result that B5 states and C8 gives the ranking for, without a number of its own. - Note: The figure for the human versus action origin, Is Human with directionality 0.99, stands only in Table XI (page 9) and in no sentence of the text. ### R55: page 10, IV-C RQ3, Results > Notably, human reviews mainly focused on these newcomers, with 79% of comments directed at them, aligning with typical community practices. - Reason: B8 would still stand, because it is about which comments are addressed, not about which authors human reviewers chose to comment on. ### R26: page 11, V Discussion > In general, human-authored review comments are much more likely to be addressed than AI-generated ones: 60% of valid human review comments led to code changes, compared to only 0.9%–19.2% for valid AI-generated comments depending on the tool (Table IX). - Repeats: B4, C4 - Reason: Repeats the result that AI-generated comments lag behind human ones, which B4 states and C4 gives the numbers for. ### R27: page 11, V-A Design Matters > Among the four actions we analyzed in depth, one stood out: coderabbitai/ai-pr-reviewer achieved a 19.2% rate of valid comments addressed, outperforming other tools by a large margin. - Reason: B4 would still stand, because it gives the range across hunk-level actions and this is the rate of the single tool at the top of that range. ### R28: page 11, V-A Design Matters > For example, mattzcarey/code-review-gpt had only 0.9% of valid comments addressed. - Reason: B4 would still stand, because it gives the range across file-level actions and this is the rate of the single tool at the bottom of that range. ### R31: page 11, V-A Design Matters > To further investigate the effectiveness of review granularity, we randomly investigated 14.4%, i.e., 30 comments, of the 208 PR-level review comments as follows: 5 from Action ID-8, 5 from ID-11, and 20 from ID-16. - Reason: Describes the study: the sample of PR-level comments inspected by hand and the pool it came from. ### R32: page 11, V-A Design Matters > Only 1 comment was partially addressed, which provided a specific performance improvement: “For better performance, preallocate the ‘architectures’ slice by using ‘make([]types.Architecture, 0, len(fm.Spec.Architectures))’ ”. - Reason: B4 would still stand, because it compares hunk-level with file-level actions and does not mention PR-level actions, which this sample is drawn from. - Note: Section V-A (page 11) uses this sample to support its conclusion that fine-grained, hunk-level reviews are more effective, so the checker can turn it into a claim if that is taken as a basis of B4. ### R42: page 11, V-A Design Matters > For the least experienced developers, the addressing rate was five times higher than for the most experienced (Table XVI). - Repeats: B8 - Reason: Repeats the contributor-experience result that B8 states, giving the comparison of the extreme bins as a ratio; R41 gives the same comparison as rates. ### R43: page 11, V-A Design Matters > For example, anc95/ChatGPT-CodeReview showed a 12.8% addressing rate for manually triggered comments versus 6.8% for automatically triggered ones (Table XIII). - Reason: B7 would still stand, because C9 gives the trigger-mode correlation over the whole sample and this is the comparison for a single action. ### R33: page 12, V-B Beyond Code Changes, Analysis of AI Comment Impact in Closed PRs > We observed a significantly lower addressing rate for AI comments in closed PRs compared to merged ones: only 3.0% (17/570) for Action ID-1, and 0% for Action ID-2 (n = 14), ID-3 (n = 18), and ID-4 (n = 0). - Reason: No broad statement states a main result about closed pull requests. - Note: Section V-B (page 12) uses this supplementary analysis to justify focusing the main results on merged pull requests. ### R34: page 12, V-B Beyond Code Changes, Analysis of Feedback on Unaddressed AI Comments > We identified a total of 50 replies and manually categorized them into four types: - Reason: Describes the study: how much material the supplementary reply analysis coded. ### R35: page 12, V-B Beyond Code Changes, Analysis of Feedback on Unaddressed AI Comments > Specifically, Willingness to Follow (10%) and Prompting Reflection (14%) imply that even if not addressed immediately, a comment may still trigger code changes in the future. - Reason: No broad statement states a main result about impact beyond immediate code changes. - Note: The conclusion (page 15) states that AI comments can still prompt developer reflection and inform future improvements, but it gives no quantity, so it is recorded as no broad statement; the checker can turn this sentence into a claim if that conclusion sentence is taken as a main result. ### R30: page 13, V-D Implications for AI Code Review Adopters > Even the best-performing AI tool achieved only a 19.2% addressing rate, far below the human reviewers’ 60%. - Repeats: B4, C4 - Reason: Repeats the result that AI-generated comments lag behind human ones, which B4 states and C4 gives the numbers for, using the top tool's rate. ### R51: page 13, VI Threats to Validity, Internal Validity > We also tried logistic regression in our experiments and it achieved a lower performance (86.1% accuracy and 0.826 Macro-F1 in 5-fold cross-validation) than our Random Forest model (88.6% and 0.846 respectively). - Reason: These are the accuracies of models that the paper only reads other findings from, compared here to justify the choice of classifier. ### R52: page 13, VI Threats to Validity, Internal Validity > To assess representativeness, the first author examined an additional 250 samples and observed similar addressing patterns as reported in our main findings. - Reason: B4 would still stand, because this is a robustness check that reports no change in the addressing patterns. ### R53: page 14, VI Threats to Validity, External Validity > As described in RQ2 Phase I, we began with 16,762 comments from 100 repositories. A language filter reduced this to 4,229 comments from 53 repositories, excluding about 75% of comments and 47 repositories. - Reason: Describes the study: the material before and after the language filter. - Note: Two passages give the count before the language filter differently. This sentence (page 14) prints 16,762 comments, while Section IV-B Phase I (page 5) prints that retaining merged-PR comments yielded 16,816 comments and that the language filter removed 12,587 of them. ### R54: page 15, VIII Conclusion and Future Work > Many AI-generated comments are not addressed, especially when they are vague or lack context. - Repeats: B4 - Reason: Repeats the result that AI-generated comments largely go unaddressed, which B4 states with numbers.