{
 "paper": {
  "id": "tse26-ai-code-review",
  "title": "Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions",
  "pdf": "evals/papers/tse26-ai-code-review.pdf",
  "pages": 17
 },
 "broad_statements": [
  {
   "id": "B1",
   "quote": "Among the 178 repositories, 82.6% customized at least one optional parameter, showing that developers often go beyond default settings when using these actions.",
   "page": 5,
   "section": "IV-A RQ1, Answer to RQ1",
   "source": "rq_answer"
  },
  {
   "id": "B2",
   "quote": "Our findings reveal that adoption is highly concentrated on four popular actions.",
   "page": 15,
   "section": "VIII Conclusion and Future Work",
   "source": "conclusion"
  },
  {
   "id": "B3",
   "quote": "We found that while adoption is growing, its effectiveness varies widely.",
   "states": "We found that adoption is growing",
   "page": 1,
   "section": "Abstract",
   "source": "abstract",
   "note": "The sentence states two main results, and B4 records the second, that effectiveness varies. B3 records the first, that adoption is growing. No claim serves it: the paper measures adoption at one point in time, for each action, across repositories, pull requests, and comments (Table IV, page 4), and reports no measurement of adoption over time."
  },
  {
   "id": "B4",
   "quote": "We found that hunk-level review actions (6.5%–19.2%) exhibit a higher addressing rate compared to file-level actions (0.9%–4.2%), yet the addressing rate of AI-generated review comments still lags behind human review comments (60%).",
   "page": 8,
   "section": "IV-B RQ2, Answer to RQ2",
   "source": "rq_answer"
  },
  {
   "id": "B5",
   "quote": "Comment addressing is strongly associated with Source Features (human vs. AI and specific action type) and Comment Features (conciseness and code-richness are better), highlighting the importance of thoughtful automated review design.",
   "page": 10,
   "section": "IV-C RQ3, Answer to RQ3",
   "source": "rq_answer"
  },
  {
   "id": "B6",
   "quote": "We introduce an LLM-assisted framework for assessing whether code review comments are actionable and whether they have been addressed, achieving high accuracy compared to human annotations.",
   "page": 2,
   "section": "I Introduction, contribution list",
   "source": "contributions"
  },
  {
   "id": "B7",
   "quote": "Comments that are concise, contain code snippets, and are manually triggered, particularly those from hunk-level review tools, are more likely to result in code changes.",
   "states": "Comments that are manually triggered are more likely to result in code changes.",
   "page": 1,
   "section": "Abstract",
   "source": "abstract",
   "note": "B5 states the conciseness and code-snippet part of this sentence and B4 the hunk-level part. Manual triggering is named by no other broad statement, so B7 is recorded for that clause. Section V-A (page 11) lists manual triggering as one of the design characteristics the study identifies."
  },
  {
   "id": "B8",
   "quote": "However, comments that are concise, specific are more likely to be addressed—particularly by less experienced contributors.",
   "states": "comments are more likely to be addressed—particularly by less experienced contributors",
   "page": 15,
   "section": "VIII Conclusion and Future Work",
   "source": "conclusion",
   "note": "B5 states the conciseness and specificity part of this sentence. The contributor-experience part is named by no other broad statement, so B8 is recorded for it. Section V-A (page 11) lists it as one of the design characteristics, under the heading Focus on newcomers."
  }
 ],
 "claims": [
  {
   "id": "C1",
   "quote": "Actions ID-1 to ID-4 (including two file-level and two hunk-level [...] reviewers) accounted for 91.1% of the reviewed repositories, 95.2% of the pull requests, and 98.9% of the generated comments.",
   "states": "Actions ID-1 to ID-4 (including two file-level and two hunk-level reviewers) accounted for 91.1% of the reviewed repositories, 95.2% of the pull requests, and 98.9% of the generated comments.",
   "page": "4-5",
   "section": "IV-A RQ1, Results",
   "serves": [
    "B2"
   ],
   "split_from": null,
   "selection_reason": "B2 says adoption is highly concentrated on four popular actions. If these four actions had not accounted for the large majority of reviewed repositories, pull requests, and comments, B2 would fail.",
   "note": "The sentence runs across the page break from page 4 to page 5 with Table V printed between its lines, which is what the [...] marks. Table IV (page 4) prints the per-action counts of repositories, pull requests, and comments, but not these three percentages."
  },
  {
   "id": "C2",
   "quote": "Regarding configuration evolution, the majority of the mature repositories (147/178, 82.6%) customized at least one optional parameter.",
   "states": "Regarding configuration evolution, the majority of the mature repositories (147/178, 82.6%) customized at least one optional parameter.",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "serves": [
    "B1"
   ],
   "split_from": null,
   "selection_reason": "B1 says 82.6% of the 178 repositories customized at least one optional parameter, showing that developers go beyond default settings. If that share were not a majority, B1 would fail."
  },
  {
   "id": "C3",
   "quote": "Overall, hunk-level review actions exhibit a higher addressing rate (6.5%–19.2%) compared to file-level actions (0.9%–4.2%).",
   "states": "Overall, hunk-level review actions exhibit a higher addressing rate (6.5%–19.2%) compared to file-level actions (0.9%–4.2%).",
   "page": 8,
   "section": "IV-B RQ2, Results",
   "serves": [
    "B4"
   ],
   "split_from": null,
   "selection_reason": "B4 says hunk-level review actions have a higher addressing rate than file-level actions. If that ordering reversed, B4 would fail.",
   "note": "Table IX (page 7) prints the per-action shares of Valid, Addressed comments that these ranges cover: 4.2% for ID-1, 0.9% for ID-2, 19.2% for ID-3, and 6.5% for ID-4."
  },
  {
   "id": "C4",
   "quote": "Beyond this discrepancy, both tables indicate that the addressing rate of AI-generated review comments (0.9%–19.2% in Table IX) still lags behind human review comments (60%).",
   "states": "Beyond this discrepancy, both tables indicate that the addressing rate of AI-generated review comments (0.9%–19.2% in Table IX) still lags behind human review comments (60%).",
   "page": 8,
   "section": "IV-B RQ2, Results",
   "serves": [
    "B4"
   ],
   "split_from": null,
   "selection_reason": "B4 says the addressing rate of AI-generated comments lags behind human review comments. If the AI rates had not been below the human rate, B4 would fail.",
   "note": "Table IX (page 7) prints 60.0% as the Valid, Addressed share for the 1,166 human comments."
  },
  {
   "id": "C5",
   "quote": "Importantly, performance on the primary downstream distinction (“Valid, Addressed” vs. all others) remains high, with an average of 91.5% overall accuracy and 89.8% macro-F1.",
   "states": "Importantly, performance on the primary downstream distinction (“Valid, Addressed” vs. all others) remains high, with an average of 91.5% overall accuracy and 89.8% macro-F1.",
   "page": 7,
   "section": "IV-B RQ2, Results",
   "serves": [
    "B6"
   ],
   "split_from": null,
   "selection_reason": "B6 presents the LLM-assisted framework as a contribution and says it achieves high accuracy compared to human annotations. This is the figure for the Valid, Addressed versus all others distinction that the RQ3 analysis rests on. If it were not high, B6 would fail.",
   "note": "This figure is measured on the 150 manually annotated comments and averaged across the three comment sources; Table VII (page 7) prints 92.0%, 91.6%, and 90.8% for the three sources and 91.5% as the average. C6 is the figure for the same distinction measured on the whole dataset."
  },
  {
   "id": "C6",
   "quote": "Considering the natural class imbalance in real-world review data, the overall classification accuracy across the full dataset is 97%, ensuring the reliability of our subsequent large-scale analysis in Phase IV (detailed breakdown in Table VIII).",
   "states": "Considering the natural class imbalance in real-world review data, the overall classification accuracy across the full dataset is 97%, ensuring the reliability of our subsequent large-scale analysis in Phase IV (detailed breakdown in Table VIII).",
   "page": 8,
   "section": "IV-B RQ2, Results",
   "serves": [
    "B6"
   ],
   "split_from": null,
   "selection_reason": "B6 presents the LLM-assisted framework as a contribution and says it achieves high accuracy compared to human annotations. This is the figure for the same Valid, Addressed versus all others distinction measured on the full dataset that Phase IV labels. If it were not high, B6 would fail.",
   "note": "This figure is measured on all 5,652 comments. Table VIII (page 7) prints Total Overall 97.0%, made up of 100% for the 3,691 comments with no code change and 91.3% for the 1,961 with code change. C5 is the figure for the same distinction measured on the annotated sample."
  },
  {
   "id": "C7",
   "quote": "The negative correlation for Is File Level Action (ρ = −0.95) aligns with the trend in Table IX, where hunk-level review actions generally outperform file-level counterparts.",
   "states": "The negative correlation for Is File Level Action (ρ = −0.95) aligns with the trend in Table IX, where hunk-level review actions generally outperform file-level counterparts.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "serves": [
    "B5"
   ],
   "split_from": null,
   "selection_reason": "B5 rests on the Source Feature results together, and this is the one for the specific action type that B5 names. If the correlation for Is File Level Action were not negative, that basis of B5 would disappear.",
   "note": "Table XI (page 9) prints Is File Level Action at rank 3 with importance 0.0346 and directionality -0.95."
  },
  {
   "id": "C8",
   "quote": "Notably, “Source Features” and “CommentP        Features” rank first and second in total importance ( |ϕ|), suggesting that both who provides the feedback (source credibility) and what the feedback contains (content quality) are strongly associated with whether comments are addressed.",
   "states": "Notably, “Source Features” and “CommentP        Features” rank first and second in total importance ( |ϕ|), suggesting that both who provides the feedback (source credibility) and what the feedback contains (content quality) are strongly associated with whether comments are addressed.",
   "page": 9,
   "section": "IV-C RQ3, Results",
   "serves": [
    "B5"
   ],
   "split_from": null,
   "selection_reason": "B5 says comment addressing is strongly associated with Source Features and Comment Features. If these two groups had not ranked first and second in total SHAP importance, B5 would fail.",
   "note": "The extraction garbled the sentence: the summation sign of the total SHAP importance came out inside the word as “CommentP        Features” and again as the space before |ϕ|. Table XI (page 9) prints the group totals 0.1589 for Source Features and 0.0959 for Comment Features."
  },
  {
   "id": "C9",
   "quote": "Automatic triggers also show a negative correlation with addressing (Trigger auto, ρ = −0.96).",
   "states": "Automatic triggers also show a negative correlation with addressing (Trigger auto, ρ = −0.96).",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "serves": [
    "B7"
   ],
   "split_from": null,
   "selection_reason": "B7 says manually triggered comments are more likely to result in code changes. If the correlation for automatic triggering were not negative, B7 would fail.",
   "note": "Table XI (page 9) prints Trigger auto at rank 2 with importance 0.0410 and directionality -0.96. Table XIII (page 10) gives the addressing rates by trigger mode: 9.0% of 2,886 automatically triggered comments and 14.4% of 104 manually triggered ones, with p > 0.05 for the total and p <= 0.05 for ID-1 and ID-2."
  },
  {
   "id": "C10",
   "quote": "For Comment Features, we find the comments were more likely to be addressed if they were concise (Text Length, ρ = −0.24) or contained a higher proportion of code (Code Text Ratio, ρ = 0.89), especially when they included multi-line code blocks(Has Multiline Code, ρ = 0.67 not shown in the table).",
   "states": "For Comment Features, we find the comments were more likely to be addressed if they were concise (Text Length, ρ = −0.24).",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "serves": [
    "B5"
   ],
   "split_from": "S1",
   "selection_reason": "B5 names conciseness as one of the Comment Features that comment addressing is associated with. If the correlation for Text Length were not negative, that part of B5 would fail.",
   "note": "Table XI (page 9) prints Text Length at rank 7 with importance 0.0133 and directionality -0.24, which is below the |ρ| > 0.3 threshold the paper states on page 9 for a meaningful correlation."
  },
  {
   "id": "C11",
   "quote": "For Comment Features, we find the comments were more likely to be addressed if they were concise (Text Length, ρ = −0.24) or contained a higher proportion of code (Code Text Ratio, ρ = 0.89), especially when they included multi-line code blocks(Has Multiline Code, ρ = 0.67 not shown in the table).",
   "states": "For Comment Features, we find the comments were more likely to be addressed if they contained a higher proportion of code (Code Text Ratio, ρ = 0.89), especially when they included multi-line code blocks(Has Multiline Code, ρ = 0.67 not shown in the table).",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "serves": [
    "B5"
   ],
   "split_from": "S1",
   "selection_reason": "B5 names code-richness as one of the Comment Features that comment addressing is associated with. If the correlation for Code Text Ratio were not positive, that part of B5 would fail.",
   "note": "Table XI (page 9) prints Code Text Ratio at rank 6 with importance 0.0155 and directionality 0.89. The sentence itself says the Has Multiline Code figure is not in the table, and Table XI does not print it."
  },
  {
   "id": "C12",
   "quote": "Within Modification Features, review comments targeting commits by experienced contributors were less likely to be addressed (Author Prior Commits, ρ = −0.69), suggesting that AI-generated reviews may be more helpful for newcomers.",
   "states": "Within Modification Features, review comments targeting commits by experienced contributors were less likely to be addressed (Author Prior Commits, ρ = −0.69), suggesting that AI-generated reviews may be more helpful for newcomers.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "serves": [
    "B8"
   ],
   "split_from": null,
   "selection_reason": "B8 says comments are more likely to be addressed by less experienced contributors. If the correlation for Author Prior Commits were not negative, B8 would fail.",
   "note": "Table XI (page 9) prints Author Prior Commits at rank 4 with importance 0.0235 and directionality -0.69."
  }
 ],
 "rejected": [
  {
   "id": "R1",
   "quote": "We present a large-scale empirical study of 16 popular AI-based code-review actions for GitHub workflows, analyzing more than 22,000 review comments in 178 repositories.",
   "page": 1,
   "section": "Abstract",
   "reason": "Every number in this summary sentence describes the study: the actions studied, the comments analyzed, and the repositories they come from."
  },
  {
   "id": "R2",
   "quote": "We develop a two-stage LLM-assisted framework to determine whether review comments are addressed.",
   "page": 1,
   "section": "Abstract",
   "duplicate_of": [
    "B6"
   ],
   "reason": "Repeats the framework that B6 states as a contribution, without the accuracy that B6 gives."
  },
  {
   "id": "R3",
   "quote": "We analyzed 178 mature repositories and found a total of 22,326 AI-generated review comments.",
   "page": 5,
   "section": "IV-A RQ1, Answer to RQ1",
   "reason": "Both numbers in this boxed sentence describe the study: the repositories analyzed and the comments collected from them."
  },
  {
   "id": "R4",
   "quote": "From a review of approximately 240 actions, we identified an initial set of 20 candidates related to AI-driven code review.",
   "page": 3,
   "section": "III Study Setup, Action Selection",
   "reason": "Describes the study: how many Marketplace actions were screened and how many were shortlisted."
  },
  {
   "id": "R5",
   "quote": "Out of 718 matched repositories, 178 met the maturity criterion (≥ 50 PRs), and these contained a total of 22,326 AI-generated review comments.",
   "page": 4,
   "section": "IV-A RQ1, Results",
   "reason": "Describes the study: the repositories matched and retained and the comments they contain, in more detail than the summary sentences give."
  },
  {
   "id": "R6",
   "quote": "We found that 37.1% of the mature repositories declared an action but showed no generated comments, indicating a gap between declaration and actual use.",
   "page": 4,
   "section": "IV-A RQ1, Results",
   "reason": "No broad statement states a main result about the gap between declaring an action and generating comments.",
   "note": "B1 states the configuration result and B2 the concentration result, and neither mentions this gap. Section IV-A (page 4) builds on it only to introduce the concentration sentence that C1 gives the numbers for, so the checker can turn it into a claim if the gap is taken as a main result."
  },
  {
   "id": "R7",
   "quote": "Among the remaining repositories, usage was highly concentrated on the four most popular actions.",
   "page": 4,
   "section": "IV-A RQ1, Results",
   "duplicate_of": [
    "B2",
    "C1"
   ],
   "reason": "Repeats the concentration result that B2 states, without a number of its own; C1 is the next sentence and gives the numbers."
  },
  {
   "id": "R8",
   "quote": "The most frequently configured options were Prompt Context Augmentation (103/147, 70.1%), primarily for specifying the natural language of review comments, LLM Selection (95, 64.6%), and Input Settings (70, 47.6%).",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "reason": "B1 would still stand, because it says that 82.6% customized at least one optional parameter, and a different ranking of which options they chose leaves that share unchanged."
  },
  {
   "id": "R9",
   "quote": "On average, these 178 repositories made changes to the configuration file 4 times after the initial setup.",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "reason": "B1 would still stand, because it is about customizing at least one parameter, not about how often the configuration file changed afterwards."
  },
  {
   "id": "R10",
   "quote": "Of these, 68 repositories (38.2%) retained their original configurations without further changes.",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "reason": "B1 would still stand, because a repository that never changed its configuration again may still have customized a parameter at setup, which is what B1 counts."
  },
  {
   "id": "R11",
   "quote": "Temporal analysis revealed that 50 projects (28.1%) finalized modifications within one week, and 20 (11.2%) within one month, while 40 repositories (22.5%) continued adjustments beyond one month.",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "reason": "B1 would still stand, because it says nothing about when repositories stopped adjusting their configurations."
  },
  {
   "id": "R12",
   "quote": "The most frequently modified parameters were LLM Selection (25, 22.7% of 110 repositories that made post-setup changes), Prompt Customization (21, 19.1%), and LLM Hyperparameters (13, 11.8%).",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "reason": "B1 would still stand, because a different ranking of the parameters changed after setup leaves the share that customized at least one parameter unchanged."
  },
  {
   "id": "R13",
   "quote": "Beyond action-specific options, we also observe that 50 repositories (45.5%) adjusted their workflow-level triggers by modifying the on or if conditions.",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "reason": "B1 would still stand, because it counts repositories that customized an optional action parameter, not repositories that edited workflow-level triggers."
  },
  {
   "id": "R14",
   "quote": "Additionally, 12 repositories (10.9%) updated the action reference, such as replacing @main with a specific release tag.",
   "page": 5,
   "section": "IV-A RQ1, Results",
   "reason": "B1 would still stand, because updating which version of an action is referenced is not customizing an optional parameter."
  },
  {
   "id": "R15",
   "quote": "After these filters, only Actions ID-1 to ID-4 retained sufficient data for the addressing analysis, with 2,908, 399, 713, and 169 comments respectively.",
   "page": 5,
   "section": "IV-B RQ2, Phase I",
   "reason": "Describes the study: how much material survived the filters for the addressing analysis.",
   "note": "The text and Table V give the count for Action ID-2 differently. This sentence (page 5) prints 399 for ID-2, while Table V (page 5) prints Total Comments 773 for mattzcarey/code-review-gpt."
  },
  {
   "id": "R16",
   "quote": "In total, our comment addressing dataset includes 5,652 review comments: 3,604 from file-level review actions, 882 from hunk-level actions, and 1,166 authored by human reviewers.",
   "page": 6,
   "section": "IV-B RQ2, Phase I",
   "reason": "Describes the study: the size of the dataset the RQ2 results rest on, split into its three kinds of comment."
  },
  {
   "id": "R17",
   "quote": "Specifically, we randomly sampled 50 comments from each category where subsequent file modifications occurred (i.e. 467 eligible from file-level actions, 424 eligible from hunk-level actions, and 1,070 eligible human-authored comments), yielding 150 comments in total.",
   "page": 6,
   "section": "IV-B RQ2, Phase II",
   "reason": "Describes the study: the sample that was annotated by hand and the pool it was drawn from."
  },
  {
   "id": "R18",
   "quote": "Inter-rater agreement (Cohen’s κ) reached substantial levels over the full 6-class annotation scheme (i.e., “None”, “General”, “Valid-Uncertain”, “Valid-Unaddressed”, “Valid-Partially”, “Valid-Fully”): 0.674 for file-level actions, 0.734 for hunk-level actions, and 0.764 for human reviews.",
   "page": 6,
   "section": "IV-B RQ2, Phase II",
   "reason": "Describes the study: an agreement score that supports the annotation procedure behind the ground truth."
  },
  {
   "id": "R19",
   "quote": "The top performers for Stage-1 were gpt-4.1 (94.5% average overall accuracy), deepseek-v3 (94.0%), and claude-3-sonnet (93.2%).",
   "page": 7,
   "section": "IV-B RQ2, Results",
   "reason": "B6 would still stand; these are per-model figures from the round that chose the setup, and C5 and C6 are the figures the later analysis rests on."
  },
  {
   "id": "R20",
   "quote": "For Stage-2, deepseek-r1 led with 95.4%, followed by openai-o3-mini (92.4%) and o4-mini (91.3%).",
   "page": 7,
   "section": "IV-B RQ2, Results",
   "reason": "B6 would still stand; these are per-model figures from the round that chose the setup, and C5 and C6 are the figures the later analysis rests on."
  },
  {
   "id": "R21",
   "quote": "Table VII reports the performance of our optimal cross-combined setup: gpt-4.1 for Stage-1 and o3-mini for Stage-2, which achieved strong and balanced performances across three comment sources, with an average of 86.1% overall accuracy and 74.6% Macro-F1 score under the full 6-class scheme.",
   "page": 7,
   "section": "IV-B RQ2, Results",
   "reason": "B6 would still stand; this is the figure for the full 6-class scheme, while C5 is the figure for the Valid, Addressed versus all others distinction that the later analysis rests on.",
   "note": "The setup this sentence names, gpt-4.1 for Stage-1 and o3-mini for Stage-2, is the one C5 and C6 report on."
  },
  {
   "id": "R22",
   "quote": "The lower macro-F1 for human-authored comments (62.4%) reflects stronger class imbalance within that subset (see Table VI), which penalizes macro-averaged metrics.",
   "page": 7,
   "section": "IV-B RQ2, Results",
   "reason": "B6 would still stand; this is the figure for one comment source within the annotated sample, where C5 gives the average across the three sources."
  },
  {
   "id": "R23",
   "quote": "Furthermore, when analyzing the complete dataset, we found that a significant portion of comments (87.0% of file-level, 51.9% of hunk-level, and 8.2% of human comments) had no subsequent modifications.",
   "page": "7-8",
   "section": "IV-B RQ2, Results",
   "reason": "Describes the study: the share of the dataset in which the reviewed file was never changed afterwards."
  },
  {
   "id": "R24",
   "quote": "In contrast, the automated annotation was applied to the entire comment dataset, which included a large volume of comments (65.3%) with no subsequent file changes.",
   "page": 8,
   "section": "IV-B RQ2, Results",
   "reason": "Describes the study: the share of the labeled dataset with no subsequent file change, given here for the dataset as a whole."
  },
  {
   "id": "R25",
   "quote": "Following the best-performing setup on the annotated dataset, we used gpt-4.1 to assess the validity of all 5,652 comments, identifying 3,955 as valid, and then used openai-o3-mini to assess whether these valid comments had been addressed.",
   "page": 8,
   "section": "IV-B RQ2, Results",
   "reason": "Describes the study: how much material the framework labeled and how much of it it judged valid."
  },
  {
   "id": "R26",
   "quote": "In general, human-authored review comments are much more likely to be addressed than AI-generated ones: 60% of valid human review comments led to code changes, compared to only 0.9%–19.2% for valid AI-generated comments depending on the tool (Table IX).",
   "page": 11,
   "section": "V Discussion",
   "duplicate_of": [
    "B4",
    "C4"
   ],
   "reason": "Repeats the result that AI-generated comments lag behind human ones, which B4 states and C4 gives the numbers for."
  },
  {
   "id": "R27",
   "quote": "Among the four actions we analyzed in depth, one stood out: coderabbitai/ai-pr-reviewer achieved a 19.2% rate of valid comments addressed, outperforming other tools by a large margin.",
   "page": 11,
   "section": "V-A Design Matters",
   "reason": "B4 would still stand, because it gives the range across hunk-level actions and this is the rate of the single tool at the top of that range."
  },
  {
   "id": "R28",
   "quote": "For example, mattzcarey/code-review-gpt had only 0.9% of valid comments addressed.",
   "page": 11,
   "section": "V-A Design Matters",
   "reason": "B4 would still stand, because it gives the range across file-level actions and this is the rate of the single tool at the bottom of that range."
  },
  {
   "id": "R29",
   "quote": "Specifically, coderabbitai/ai-pr-reviewer achieved the highest valid and addressed rate (19.2%).",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B4 would still stand, because it gives the range 6.5% to 19.2% across hunk-level actions and this is the rate of the single tool at the top of it."
  },
  {
   "id": "R30",
   "quote": "Even the best-performing AI tool achieved only a 19.2% addressing rate, far below the human reviewers’ 60%.",
   "page": 13,
   "section": "V-D Implications for AI Code Review Adopters",
   "duplicate_of": [
    "B4",
    "C4"
   ],
   "reason": "Repeats the result that AI-generated comments lag behind human ones, which B4 states and C4 gives the numbers for, using the top tool's rate."
  },
  {
   "id": "R31",
   "quote": "To further investigate the effectiveness of review granularity, we randomly investigated 14.4%, i.e., 30 comments, of the 208 PR-level review comments as follows: 5 from Action ID-8, 5 from ID-11, and 20 from ID-16.",
   "page": 11,
   "section": "V-A Design Matters",
   "reason": "Describes the study: the sample of PR-level comments inspected by hand and the pool it came from."
  },
  {
   "id": "R32",
   "quote": "Only 1 comment was partially addressed, which provided a specific performance improvement: “For better performance, preallocate the ‘architectures’ slice by using ‘make([]types.Architecture, 0, len(fm.Spec.Architectures))’ ”.",
   "page": 11,
   "section": "V-A Design Matters",
   "reason": "B4 would still stand, because it compares hunk-level with file-level actions and does not mention PR-level actions, which this sample is drawn from.",
   "note": "Section V-A (page 11) uses this sample to support its conclusion that fine-grained, hunk-level reviews are more effective, so the checker can turn it into a claim if that is taken as a basis of B4."
  },
  {
   "id": "R33",
   "quote": "We observed a significantly lower addressing rate for AI comments in closed PRs compared to merged ones: only 3.0% (17/570) for Action ID-1, and 0% for Action ID-2 (n = 14), ID-3 (n = 18), and ID-4 (n = 0).",
   "page": 12,
   "section": "V-B Beyond Code Changes, Analysis of AI Comment Impact in Closed PRs",
   "reason": "No broad statement states a main result about closed pull requests.",
   "note": "Section V-B (page 12) uses this supplementary analysis to justify focusing the main results on merged pull requests."
  },
  {
   "id": "R34",
   "quote": "We identified a total of 50 replies and manually categorized them into four types:",
   "page": 12,
   "section": "V-B Beyond Code Changes, Analysis of Feedback on Unaddressed AI Comments",
   "reason": "Describes the study: how much material the supplementary reply analysis coded."
  },
  {
   "id": "R35",
   "quote": "Specifically, Willingness to Follow (10%) and Prompting Reflection (14%) imply that even if not addressed immediately, a comment may still trigger code changes in the future.",
   "page": 12,
   "section": "V-B Beyond Code Changes, Analysis of Feedback on Unaddressed AI Comments",
   "reason": "No broad statement states a main result about impact beyond immediate code changes.",
   "note": "The conclusion (page 15) states that AI comments can still prompt developer reflection and inform future improvements, but it gives no quantity, so it is recorded as no broad statement; the checker can turn this sentence into a claim if that conclusion sentence is taken as a main result."
  },
  {
   "id": "R36",
   "quote": "Finally, positive correlations for Repository Features demonstrated that comments were more likely to be addressed in large and active projects (Repo File Size, ρ = 0.70; Repo Issue Count, ρ = 0.64, ranked 13/36).",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B5 would still stand, because it names Source Features and Comment Features as the groups comment addressing is associated with, not Repository Features."
  },
  {
   "id": "R37",
   "quote": "Interestingly, the LLM choice did not show high predictive importance (LLM GPT-3.5 ranked 19/36 and LLM GPT-4 ranked 21/36 in the feature list of the overall model; Is Model GPT4 ranked 23/33 in the feature list of the AI-specific model), indicating that simply employing a more advanced model does not decisively guarantee comment addressing.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B5 would still stand, because the Source Features it names are human versus AI and the specific action type, not which LLM the action used."
  },
  {
   "id": "R38",
   "quote": "However, when comparing addressing rates between two model families (Table XIV), GPT-4 generated comments did show better addressing rates.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B5 would still stand, because it names human versus AI and the specific action type as its Source Features, not the LLM family.",
   "note": "Table XIV (page 10) prints, for the total, 5.4% of 1,435 GPT-3.5 comments and 12.6% of 1,555 GPT-4 comments addressed, with p <= 0.05, while R37 on the same page reports that the LLM-choice features ranked low in predictive importance."
  },
  {
   "id": "R39",
   "quote": "Regarding topics, comments were addressed more often when they pertained to “Concurrency Control and Locking Robustness” (e.g., “using a ‘set-if-not-exists’ pattern”) (LDA Topic 2, ρ = 0.50), as well as “UI Usability and Accessibility Enhancements” (e.g., “adding aria-label for screen reader support”) (LDA Topic 1, ρ = 0.41).",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B5 would still stand, because the Comment Features it names are conciseness and code-richness, not the topic a comment belongs to."
  },
  {
   "id": "R40",
   "quote": "As shown in Table XV, the addressing rate for both human and AI-generated comments increased noticeably when Code Text Ratio exceeded 0.5.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B5 rests on the code-richness result over the whole sample, which C11 gives; this sentence gives the pattern for binned categories of Code Text Ratio.",
   "note": "Table XV (page 10) prints addressing rates across the five bins of 4.2%, 7.2%, 5.7%, 10.5%, and 23.2% for the four actions and 71.2%, 65.6%, 75.8%, 78.5%, and 89.9% for human comments."
  },
  {
   "id": "R41",
   "quote": "As shown in Table XVI, valid comments directed at project newcomers (Author Prior Commits ≤ 124) achieved 16% addressing rate, compared to just 3.3% for the most experienced contributors.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B8 rests on the contributor-experience result over the whole sample, which C12 gives; this sentence gives the rates for two of the binned experience categories.",
   "note": "The text and Table XVI give the newcomer figure differently. This sentence (page 10) prints 16%, while Table XVI (page 10) prints 16.1% for the (-0.00, 30] bin and 16.1% for the (30, 124] bin; the table prints 3.3% for the (1013, 4,316] bin."
  },
  {
   "id": "R42",
   "quote": "For the least experienced developers, the addressing rate was five times higher than for the most experienced (Table XVI).",
   "page": 11,
   "section": "V-A Design Matters",
   "duplicate_of": [
    "B8"
   ],
   "reason": "Repeats the contributor-experience result that B8 states, giving the comparison of the extreme bins as a ratio; R41 gives the same comparison as rates."
  },
  {
   "id": "R43",
   "quote": "For example, anc95/ChatGPT-CodeReview showed a 12.8% addressing rate for manually triggered comments versus 6.8% for automatically triggered ones (Table XIII).",
   "page": 11,
   "section": "V-A Design Matters",
   "reason": "B7 would still stand, because C9 gives the trigger-mode correlation over the whole sample and this is the comparison for a single action."
  },
  {
   "id": "R44",
   "quote": "We found that in the two actions with manually triggered comments (Action ID-1 and ID-2), manually triggered comments consistently showed higher addressing rates.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B7 rests on the trigger-mode result over the whole sample, which C9 gives; this sentence gives the comparison for the two actions that have manually triggered comments.",
   "note": "The sentence gives no number of its own. Table XIII (page 10) holds them: 6.8% of 1,595 automatic versus 12.8% of 86 manual comments for ID-1, and 0.5% of 602 versus 22.2% of 18 for ID-2, both with p <= 0.05."
  },
  {
   "id": "R45",
   "quote": "Source Features represent the most significant group, showing a clear difference in developers’ responses based on the comment origins (human vs. actions; file-level vs. hunk-level).",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "duplicate_of": [
    "B5",
    "C8"
   ],
   "reason": "Repeats the group-importance result that B5 states and C8 gives the ranking for, without a number of its own.",
   "note": "The figure for the human versus action origin, Is Human with directionality 0.99, stands only in Table XI (page 9) and in no sentence of the text."
  },
  {
   "id": "R46",
   "quote": "We focus specifically on the 3,879 valid comments from RQ2 that have a definitive addressing label (excluding 76 uncertain cases), examining the factors that determine whether they are addressed.",
   "page": 8,
   "section": "IV-C RQ3, Approach",
   "reason": "Describes the study: the comments the factor analysis rests on and the ones excluded from it."
  },
  {
   "id": "R47",
   "quote": "After checking for multicollinearity, we removed 9 redundant features, yielding a final set of 36 features presented in Table X.",
   "page": 8,
   "section": "IV-C RQ3, Phase I",
   "reason": "Describes the study: the size of the feature set the factor analysis uses."
  },
  {
   "id": "R48",
   "quote": "To enable focused analysis, we mapped the 3,879 comments into a binary classification task: comments labeled as “Valid-Partially” or “Valid-Fully” were grouped as the “Addressed” class (25.1%), with “Valid-Unaddressed” comments forming the “Not Addressed” class (74.9%).",
   "page": 9,
   "section": "IV-C RQ3, Phase II",
   "reason": "Describes the study: how the labeled comments were split into the two classes the classifier was trained on."
  },
  {
   "id": "R49",
   "quote": "The model achieved an average of 88.6% overall accuracy (Macro-F1=0.846) across the five test folds, indicating that our engineered features are effective predictors of comment addressing.",
   "page": 9,
   "section": "IV-C RQ3, Phase II",
   "reason": "It is the accuracy of a model that the paper only reads other findings from, namely the SHAP importances and directions that B5 rests on."
  },
  {
   "id": "R50",
   "quote": "The resulting AI-Specific Model achieved an overall accuracy of 91.7% and a Macro-F1 score of 0.647 in 5-fold cross-validation.",
   "page": 9,
   "section": "IV-C RQ3, Phase III",
   "reason": "It is the accuracy of a model that the paper only reads other findings from, namely the SHAP results for AI-generated comments."
  },
  {
   "id": "R51",
   "quote": "We also tried logistic regression in our experiments and it achieved a lower performance (86.1% accuracy and 0.826 Macro-F1 in 5-fold cross-validation) than our Random Forest model (88.6% and 0.846 respectively).",
   "page": 13,
   "section": "VI Threats to Validity, Internal Validity",
   "reason": "These are the accuracies of models that the paper only reads other findings from, compared here to justify the choice of classifier."
  },
  {
   "id": "R52",
   "quote": "To assess representativeness, the first author examined an additional 250 samples and observed similar addressing patterns as reported in our main findings.",
   "page": 13,
   "section": "VI Threats to Validity, Internal Validity",
   "reason": "B4 would still stand, because this is a robustness check that reports no change in the addressing patterns."
  },
  {
   "id": "R53",
   "quote": "As described in RQ2 Phase I, we began with 16,762 comments from 100 repositories. A language filter reduced this to 4,229 comments from 53 repositories, excluding about 75% of comments and 47 repositories.",
   "page": 14,
   "section": "VI Threats to Validity, External Validity",
   "reason": "Describes the study: the material before and after the language filter.",
   "note": "Two passages give the count before the language filter differently. This sentence (page 14) prints 16,762 comments, while Section IV-B Phase I (page 5) prints that retaining merged-PR comments yielded 16,816 comments and that the language filter removed 12,587 of them."
  },
  {
   "id": "R54",
   "quote": "Many AI-generated comments are not addressed, especially when they are vague or lack context.",
   "page": 15,
   "section": "VIII Conclusion and Future Work",
   "duplicate_of": [
    "B4"
   ],
   "reason": "Repeats the result that AI-generated comments largely go unaddressed, which B4 states with numbers."
  },
  {
   "id": "R55",
   "quote": "Notably, human reviews mainly focused on these newcomers, with 79% of comments directed at them, aligning with typical community practices.",
   "page": 10,
   "section": "IV-C RQ3, Results",
   "reason": "B8 would still stand, because it is about which comments are addressed, not about which authors human reviewers chose to comment on."
  }
 ]
}