Claim-Evidence Alignment (CEA)

Claim-Evidence Alignment (CEA) is a framework for assessing whether the evidence in a paper and its published research artifact supports the paper's claims. For each claim, CEA reconstructs the chain that connects the claim to its evidence, then compares what the claim asserts with what the chain shows.

A claim can have a complete chain and still lack support, because the chain only shows how the claim connects to the evidence. A separate alignment assessment decides whether the evidence supports the claim. A gap in the chain is recorded as a finding and is never filled with assumptions.

Scope

CEA covers narrow quantitative empirical claims, as defined in Section 1. Its sources are the published paper and the published research artifact. An agent drafts the selection of claims, each chain, and each alignment assessment, and a person, the checker, reviews them.

CEA works at the conceptual level and does not define how to find chain elements in concrete papers and artifacts. It also excludes these topics:

qualitative, formal, and novelty claims
causal graphs and theory building
sub-chains, which split a single link into finer steps
representation formats
comparisons across papers

1. Narrow claims

A narrow claim is a specific statement of a quantitative result in the paper. Statements in the abstract or the contribution list are usually broader. CEA does not reconstruct chains for them directly but uses them to find the narrow claims that they depend on.

Select a narrow claim if the answer to this question is yes:

If this statement were false or unsupported, would a main result or key contribution fail or need substantial revision?

Record why each claim was selected. Split a statement into separate claims when its parts need different evidence. For example, "X is faster and uses less memory" becomes two claims.

The selection record

A checker reviews the selection itself, not only its outcome, so the record holds all three of:

each main result of the paper, quoted from the sentence that states it, with every other place the paper states it again
each narrow claim, quoted with its location, the main results it serves, and how each of them would fail if the claim were false
each statement that a checker could expect to be a claim and that was not selected, with the ground that excludes it

Recording what was set aside, and why, is what makes the selection reviewable: a checker can overturn any of the three.

A main result that the paper states as a qualitative finding is out of scope, so the record holds no claim for it. A main result that no narrow claim serves stays in the record as a finding, with the reason the paper's own evidence does not reach it.

2. The chain

The chain runs from a narrow claim through its evidence and back to the claim.

flowchart LR
    C[Narrow claim] -->|1 interpretation| K[Claim content]
    K -->|2 operationalization| M[Measures]
    M -->|3 measurement| O[Observations]
    O -->|4 unit bridge| U[Unit-level data]
    U -->|5 analysis| R[Result]
    R -->|6 reasoning| C

Elements

Element	Meaning
Narrow claim	The claim at the start of the chain.
Claim content	The constructs and their relation, the unit of analysis, the scope, and the claim kind. The scope covers the entities, settings, and period that the claim refers to. The claim kind is descriptive, associational, predictive, or causal.
Measures	What represents each construct in a form that can be observed.
Observations	The recorded data, which describes entities at the unit of observation.
Unit-level data	Observations brought to the unit of analysis.
Result	What the analysis of the unit-level data shows.

A construct is a concept that the claim asserts something about, such as seniority, coding time, or throughput. Some measures represent their construct directly, and others only indirectly. The less directly a measure represents its construct, the more justification the reasoning link needs.

Links

#	Link	Question the link answers
1	Interpretation	What does the claim assert, about which entities, and within which scope?
2	Operationalization	Which measure represents each construct?
3	Measurement	Which observations carry each measure?
4	Unit bridge	How do observations about one kind of entity become data about the unit of analysis?
5	Analysis	How does the analysis produce the result?
6	Reasoning	Why does each measure represent its construct, why does the result bear on the claim, and which assumptions and objections apply? For a causal claim, the reasoning includes the causal assumptions.

The unit bridge is a separate link because a claim and its data can concern different units. A claim about developers that rests on data about commits needs this bridge, and aggregating commits to developers can change the relation that a result shows [8]. When the observations already describe the unit of analysis, the unit bridge needs no further step.

A narrow claim can rest on several results, each with its own chain. When the claim needs the results jointly, every chain is reconstructed. When one result alone supports the claim, reconstructing its chain is enough, and the record for the claim, described in Section 7, states why.

3. Link quality

Each link has one of these qualities:

Quality	Meaning
Clear	A checker can confirm the link from the paper or the artifact without adding assumptions.
Speculative	The link is plausible but depends on an added assumption.
Gap	No link was found, or the material needed is not accessible.

Only clear links count toward the mapping level in Section 4. A clear link counts whether the authors stated it or it was reconstructed from the artifact.

Each link also records its basis:

For a clear link, the basis says where the link can be confirmed and whether the authors stated it or it was reconstructed.
For a speculative link, the basis names the assumption that the link depends on.
For a gap, the basis names what is missing or not accessible.

No rule fixed in advance separates clear from speculative links for every claim, because the judgment depends on the field, the data, and what the checker knows. The basis makes each judgment open to inspection. The checker accepts or overturns each drafted link quality using its basis, and borderline links found during checking are collected as examples. Over time, these examples show how checkers distinguish clear from speculative links in practice.

Link quality records whether the link is established. For example, if commit counts are established as the measure of coding time, the operationalization link is clear. The authors' justification for why commit counts represent coding time belongs to the reasoning link, and the alignment assessment in Section 5 judges whether that justification is adequate.

The reasoning link is clear only when the authors made the argument, in the paper or in the artifact. A justification that the agent or the checker supplies is an added assumption, so it is recorded as speculative and does not count. The requirement that the authors made the argument applies in particular to the justification that a measure represents its construct.

4. Mapping level

The mapping level is the highest level that a chain reaches, and it records how far the chain could be reconstructed. Gaps and speculative links earlier in the chain do not lower the level. They are reported with it, so that the level and its gaps together describe what the reconstructed chain covers and where it is incomplete.

Level	The reconstructed chain reaches
M1	The claim, with its exact wording, location, and selection reason.
M2	The claim content, because link 1 is clear.
M3	The result, because link 5 is clear.
M4	The reasoning, because link 6 is clear.

Report the level for each reconstructed chain together with the links that are not clear, as in "M4, gap at the unit bridge". A high level does not indicate strong support.

5. Alignment assessment

The alignment assessment compares what the claim asserts with what the chain shows, and looks for these mismatches:

Mismatch	Meaning
Construct	A measure represents a different construct than the claim names.
Unit	The result concerns a different unit than the claim.
Scope	The observations cover fewer entities, fewer settings, or a shorter period than the claim.
Relation	The result shows a weaker, different, or opposite relation.
Claim kind	The claim is stronger in kind than the evidence, such as a causal claim resting on a descriptive comparison.

The assessment ends in one verdict:

Verdict	Meaning
Supports the claim as stated	No mismatch was found. The verdict names the limits of the support.
Supports a narrower claim	A mismatch exists, and the chain supports a narrower wording, which the verdict states.
Insufficient support	A link needed for the comparison is speculative or a gap, or the result does not establish the relation.
Conflicting evidence	The result contradicts the claim.
Not assessable	Material needed for the comparison is not accessible.

A verdict rests on the reconstructed chain, and the authors' statement that their evidence supports the claim is not enough on its own. Missing access does not show that the evidence contradicts the claim. A successful run of the analysis does not show that a measure represents the intended construct.

6. Example

The example uses a hypothetical narrow claim, "In the studied projects, senior developers spent less time coding than junior developers."

Link	Chain	Quality and basis
1 Interpretation	The constructs are seniority and coding time, the unit is the developer, the scope is the studied projects, and the claim is descriptive.	Clear, stated in the paper
2 Operationalization	Seniority is measured as project tenure, and coding time as commit counts.	Clear, reconstructed from the artifact
3 Measurement	Commit records come from the projects' version histories.	Clear, stated in the paper
4 Unit bridge	Commits are attributed to developers by author identity.	Clear, reconstructed from the artifact
5 Analysis	Commit counts are compared between senior and junior developers.	Clear, stated in the paper
6 Reasoning	Neither the paper nor the artifact argues that commit counts reflect coding time.	Gap, argument absent

If the artifact did not show how commits are attributed to developers, the unit bridge would be speculative. Its basis would name the assumption that each developer commits under a single identity.

The mapping level is "M3, gap at the reasoning link", because the chain reaches the result but not the reasoning. A justification proposed by the checker would be speculative and would leave the level at M3.

The alignment assessment finds two construct mismatches. Commit counts do not represent coding time, and project tenure does not represent seniority in general. The verdict is "supports a narrower claim", with the narrower wording "In the studied projects, developers with longer project tenure made fewer commits than developers with shorter tenure."

7. Record for each claim

The record for each narrow claim contains the following items:

the claim, its location, and the reason for selecting it
each link of the chain with its quality, its basis, and the outcome of the check
the reason for stopping when chains are reconstructed for only some of the claim's results
the mapping level of each reconstructed chain, with its gaps
the mismatches and the verdict, with reasons

8. Foundations

CEA combines established ideas, and its individual concepts are not new. Its proposed contribution is a procedure for reconstructing the chain behind a narrow quantitative claim from a finished paper and artifact, and for judging whether that chain supports the claim. A study still has to show that the procedure is useful, for example by testing whether independent checkers agree on claim selection, link qualities, and verdicts. Micropublications and SEE describe arguments without evaluating them [1, 2]. CEA adds records of gaps and a verdict for each claim.

Work	Contribution	Use or extension in CEA
Micropublications [1]	Statements with attribution, data, methods, and challenges, in forms that range from a statement with its attribution to the statement with its complete supporting argument. Support is a single transitive relation	Checks each link separately, so a valid computation does not establish that a measure represents its construct. Adds construct, unit, and scope links for quantitative claims, and alignment verdicts
SEE [2]	Scientific claims, their subjects, and consecutive layers of interpretation and attribution, including a curator's evaluation of a report. Premises used jointly, measurement statements represented as assertions, and alternative interpretations that infer a less specific conclusion from the same data. A distinction between a curator's conclusion based on an author's statement and a conclusion inferred from the reported data	Separates links that authors stated from links that others reconstructed. Reconstructs every chain when results are needed jointly, treats a narrower claim like an alternative interpretation, and bases verdicts on the reconstructed chain
Nomological networks [3] and measurement in software engineering [4, 5]	A network of laws that relates constructs to each other and to observables [3]. Measurement models that operationalize constructs through indicators [4]. Construct validity as the adequacy of a concept definition and of the indicators that represent it [5]	Applies these ideas to the chain of a single claim
Evidence-Centered Design [6]	An approach to designing educational assessments around the inferences to be made, the observations that ground them, and the chain of reasoning that connects them	Applies the same connections to finished research instead of assessment design
Estimand framework [7]	A theoretical estimand, defined by a unit-specific quantity and a target population, linked to an empirical estimand under identification assumptions and then learned from data	Works backward, reconstructing what was estimated and comparing it with the claim
Simpson's paradox [8]	A practical guide showing that an association in a population can reverse within its subgroups, most likely when inferences cross levels of explanation	Motivates the unit bridge as a separate link
Classification of data science tasks [9]	Description, prediction, and counterfactual prediction, which includes causal inference, as distinct classes of tasks	Motivates the claim kind mismatch
SciFact [10]	A task and dataset for finding research abstracts with evidence that supports or refutes a scientific claim, with rationales that justify each decision	Ties verdicts to a reconstructed chain and adds a verdict for narrower claims
Toulmin's argument model [11] and OntoGSN [12]	What an argument states, how it is qualified and backed, and what may contradict it [11]. Goals supported by strategies and evidence, with context, assumptions, and justifications [12]	Informs the reasoning link
PROV [13] and FAIRSCAPE [14]	A data model for the entities, people, and processes involved in producing data [13]. Evidence graphs that the FAIRSCAPE framework creates for each computational result, linking it to the software, computations, and datasets used [14]	Reconstructs the chain after publication and asks whether the results address the claim

References

T. Clark, P. N. Ciccarese, and C. A. Goble. Micropublications: a semantic model for claims, evidence, arguments and annotations in biomedical communications. Journal of Biomedical Semantics 5:28, 2014. <https://doi.org/10.1186/2041-1480-5-28>
C. Bölling, M. Weidlich, and H.-G. Holzhütter. SEE: structured representation of scientific evidence in the biomedical domain using Semantic Web techniques. Journal of Biomedical Semantics 5(Suppl 1):S1, 2014. <https://doi.org/10.1186/2041-1480-5-S1-S1>
L. J. Cronbach and P. E. Meehl. Construct validity in psychological tests. Psychological Bulletin 52(4):281-302, 1955. <https://doi.org/10.1037/h0040957>
P. Ralph, M. Kuutila, H. Arif, and B. Ayoola. Teaching Software Metrology: The Science of Measurement for Software Engineering. arXiv:2406.14494, 2024. <https://arxiv.org/abs/2406.14494>
D. I. K. Sjøberg and G. R. Bergersen. Improving the Reporting of Threats to Construct Validity. EASE, 2023. <https://doi.org/10.1145/3593434.3593449>
R. J. Mislevy, R. G. Almond, and J. F. Lukas. A Brief Introduction to Evidence-Centered Design. CSE Report 632, CRESST, 2004. <https://cresst.org/wp-content/uploads/R632.pdf>
I. Lundberg, R. Johnson, and B. M. Stewart. What Is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory. American Sociological Review, 2021. <https://doi.org/10.1177/00031224211004187>
R. A. Kievit, W. E. Frankenhuis, L. J. Waldorp, and D. Borsboom. Simpson's paradox in psychological science: a practical guide. Frontiers in Psychology 4:513, 2013. <https://doi.org/10.3389/fpsyg.2013.00513>
M. A. Hernán, J. Hsu, and B. Healy. A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks. CHANCE 32(1):42-49, 2019. <https://doi.org/10.1080/09332480.2019.1579578>
D. Wadden et al. Fact or Fiction: Verifying Scientific Claims. EMNLP, 7534-7550, 2020. <https://aclanthology.org/2020.emnlp-main.609/>
S. E. Toulmin. The Uses of Argument. Cambridge University Press, 1958, updated edition 2003.
T. Bueno Momcilovic, B. Gallina, I. Kessler, and D. Balta. OntoGSN: An Ontology for Dynamic Management of Assurance Cases. arXiv:2506.11023, 2025. <https://arxiv.org/abs/2506.11023>
Y. Gil and S. Miles, editors. PROV Model Primer. W3C Working Group Note, 2013. <https://www.w3.org/TR/2013/NOTE-prov-primer-20130430/>
M. A. Levinson et al. FAIRSCAPE: a Framework for FAIR and Reproducible Biomedical Analytics. Neuroinformatics 20:187-202, 2022. <https://doi.org/10.1007/s12021-021-09529-4>