You compare claims of a published review (the target) with a generated review. You never see how the generated review was produced; you judge only the text.
Target claims are listed one per line with an id (from the abstract's results and conclusions and, when present, the plain language key messages). For each target claim decide a label. Direction categories are: benefit, harm, little or no difference, uncertain.
| label | meaning |
|---|---|
matched | the generated review states the same finding: same comparison and outcome, same direction category, and the same certainty level (high "results in", moderate "probably", low "may", very low "very uncertain"); a numeric estimate, if the target gives one, agrees to the stated precision |
partial | same comparison and outcome, and either the same direction with the certainty one level apart, an estimate beyond rounding or a missing qualifier (population, time point), or a neighbouring direction category (benefit or harm against "little or no difference"; any direction against "uncertain") |
missing | the generated review makes no statement about this comparison and outcome |
contradicted | the generated review states the opposite direction (benefit against harm), or a certainty two or more levels apart, or a conclusion the target explicitly rejects in words (a neighbouring category alone is not such a rejection) |
Two conventions for one imprecise finding (addendum 5, A11). When the 95% confidence interval includes or reaches no effect (1 for ratios, 0 for differences), as shown in the target claim or the generated sentence, reviews word the same finding in two ways: a hedged direction from the point estimate ("may be slightly higher", "likely results in a slight reduction") or "little or no difference". Treat these as the same direction category when the estimate agrees to the stated precision and the certainty level is the same; label by the other conditions, set convention_equivalent to true, and name the interval in note. "Uncertain" against a stated direction stays a neighbouring category: very-low-certainty wording carries no direction. Otherwise set convention_equivalent to false.
A claim with several findings takes the label of its worst part. Note the parts in note.
Information the generated review could not have (stage-isolated runs; listed below the claims as "SI input"). If a claim, or part of one, needs information that is not in that input, set input_limited to part or all. Quote that part verbatim from the target claim in input_limited_text, and choose one input_limited_category:
study_status_or_search: counts or lists of excluded, ongoing or awaiting studies, or search and screening facts not in the protocol input;review_process: update schedules or plans not in the protocol input;certainty_reasons: why certainty was rated as it was (levels themselves are in the input);results_outside_analyses: findings that are in no analysis listed in the input.
For part, label only the remainder. For all, still give your best label. Otherwise set input_limited to none.
Target deviations. The target's own Summary of findings ratings are listed below the claims. If a target claim's wording implies a certainty level for a comparison and outcome that differs from the target's own rating there, set target_deviation to the implied level and td_row to the listed row's id. An unhedged "are effective" or "reduces" implies high; "likely" or "probably" implies moderate; "may" implies low; "very uncertain" implies very low. Otherwise set target_deviation to null.
Rules: judge meaning, not wording. A target claim about "no studies found" for an outcome is matched by a generated statement that no studies reported it. Quote the generated sentence you used (up to 40 words) for every label except missing. Output JSON only:
{"claims": [{"id": "T1", "label": "matched|partial|missing|contradicted", "generated_quote": "...", "note": "...", "input_limited": "none|part|all", "input_limited_text": "", "input_limited_category": "", "target_deviation": null, "td_row": null, "convention_equivalent": false}]}