A systematic review from question to written review, each stage measured against published Cochrane reviews.
Edition 2.0.0 starts where a review starts. It drafts and checks the protocol from the question, builds and runs concept-block searches of PubMed, Europe PMC and ClinicalTrials.gov with a PRISMA-S record, de-duplicates, screens titles and abstracts and then full texts in two independent passes with adjudication, links reports to studies, extracts data twice with verbatim quotes and reconciles, plans and runs the analyses with the 1.0.0 engines, records GRADE decisions, and writes the complete review in the Cochrane 2024 structure with every number filled from a results file. Each stage is measured by blind replication of six Cochrane reviews published after the model's training cutoff. The edition is a release candidate: its acceptance contract's release gate has not been passed, and this page shows which thresholds have been met.
$5.00 one-off
Preview — package delivery is not available yet
R 4.3 or newer with meta 8.x, metafor, netmeta, bayesmeta, jsonlite and ggplot2 (qualified with R 4.3.2, meta 8.5-0, metafor 4.6-0, netmeta 3.7-0, bayesmeta 3.5). Dependencies are installed separately under their own GPL and MIT licences.
Owner-authorized and AI-assisted. Package identity, a clean-extraction replay of both test suites, the render check, the blind replication record and the showcase are recorded; no independent human statistical review, clinical certification or legal-counsel review is claimed. The 2.0.0 release gate (two consecutive passing iterations on the development reviews, then locked runs on unseen reviews) has not been passed.
Agreement with a published Cochrane review is a replication result: it does not certify any clinical conclusion, and the review team's judgements (eligibility, risk of bias, certainty) remain theirs to make. The search and retrieval scripts call public services under their terms and never attempt paywalled or bot-protected routes; databases they cannot reach (Embase, CENTRAL, PsycINFO, WHO ICTRP) need a person with access. The Cochrane Library is not queried.
The 2.0.0 acceptance contract was frozen before the review stages were built and amended five times with the owner's recorded approval. Six Cochrane reviews published after the model's training cutoff form the development set. Each stage runs blind from the review's own protocol, with the other stages replaced by the review's published records (stage-isolated modes), and its output is scored against the published review by a deterministic harness; prose claims are judged by two independent judges with adjudication.
The release gate requires every threshold to be met in two consecutive iterations on the development set, then locked runs on reviews never used for development. Fifteen iterations are logged with their costs. The table gives the latest measured value of every metric; where a threshold is not met or a stage has not run, it says so.
| Metric | Mode | Development threshold | Latest, pooled | Reviews run |
|---|---|---|---|---|
| Search recall, accessible reports | SI-search | ≥ 95% | 8/8 (100%) | 2 |
| Search recall, all included studies | SI-search | ≥ 85% | 8/8 (100%) | 2 |
| Search documentation (PRISMA-S) | SI-search | 100% | 2/2 (100%) | 2 |
| Title/abstract sensitivity | SI-screen | ≥ 98% | 7/7 (100%) | 1 |
| Inclusion sensitivity (full text) | SI-screen | ≥ 95% | not run | 0 |
| Inclusion precision | SI-screen | ≥ 85% | not run | 0 |
| Exclusion agreement | SI-screen | ≥ 90%; ≥ 70% | not run | 0 |
| Status agreement | SI-screen | ≥ 70% | not run | 0 |
| Report-to-study linkage (F1) | SI-extract | ≥ 0.95; ≥ 90% | mean 0.86 | 6 |
| Study labels | SI-extract | ≥ 0.95; ≥ 90% | 43/53 (81%) | 6 |
| Extracted data rows exact | SI-extract | ≥ 90% | 83/115 (72%) | 6 |
| Analysis structure | SI-extract | ≥ 85% | 78/102 (76%) | 6 |
| Characteristics populated | SI-extract | ≥ 95%; ≥ 90% | 244/244 (100%) | 6 |
| Risk-of-bias domains exact | SI-extract | ≥ 70% / ≥ 70% | 82/136 (60%) | 6 |
| Certainty level exact | SI-certainty | ≥ 60%; ≥ 90% | 25/41 (61%) | 6 |
| Certainty within one level | SI-certainty | ≥ 60%; ≥ 90% | 39/41 (95%) | 6 |
| Numbers traced to results | SI-write | 100%; ≥ 95% | 1940/1940 (100%) | 6 |
| Target estimates reproduced | SI-write | 100%; ≥ 95% | 35/35 (100%) | 6 |
| Summary of findings rows present | SI-write | 100%; ≥ 90% | 41/41 (100%) | 6 |
| Abstract claims matched (contradicted) | SI-write | ≥ 90%; 0 | 48/51 (94%); 0 contradicted | 6 |
| Verbatim 12-word overlaps | SI-write | 0 | 4 over 6 reviews | 6 |
| Process integrity | SI-write | pass | 6/6 pass | 6 |
Target: Butler AR, Lindson N, Livingstone-Banks J, et al., Interventions for quitting vaping, Cochrane Database of Systematic Reviews 2025, CD016058.pub2 (CC BY 4.0). The writing stage received the review's protocol text, its included studies with their characteristics, its analysis data and engine results, its risk-of-bias judgements and the certainty levels of its Summary of findings rows (without footnotes). It did not receive the review's abstract or text. It wrote the complete review; every number was filled from a results file.
Two independent judges compared the sentences of the published abstract (results and conclusions) with the generated review under the rubric, and a third adjudicated the one claim on which they differed. The table lists every claim with its label; the generated review is published in full.
| Claim | Section | Published claim | Label | Marks |
|---|---|---|---|---|
| T1 | Synthesis of results | Synthesis of resultsPharmacological interventions for quitting nicotine vapingStudies assessed combination nicotine replacement therapy (NRT), cytisine, and var… | matched | — |
| T2 | Synthesis of results | The point estimate for combination NRT indicated possible benefit, but the CI incorporated the possibility of no benefit and a potential benefit of control (ver… | matched | input-limited (certainty_reasons) |
| T3 | Synthesis of results | The one study investigating cytisine did not report vaping cessation rates at six months or longer. | matched | — |
| T4 | Synthesis of results | Varenicline increased vaping cessation rates at six months, but the evidence was low certainty due to imprecision (RR 2.00, 95% CI 1.09 to 3.68; 1 study, 140 pa… | matched | input-limited (certainty_reasons) |
| T5 | Synthesis of results | However, only one study reported an SAE (in the intervention arm); therefore, the effect estimate was calculated based on that single study (RR 2.60, 95% CI 0.1… | matched | input-limited (certainty_reasons); A11 |
| T6 | Synthesis of results | In one study, the point estimate suggested nicotine/vaping reduction increased vaping cessation compared to minimal support at six‐month follow‐up, but the CI i… | matched | input-limited (certainty_reasons) |
| T7 | Synthesis of results | There was low‐certainty evidence (downgraded two levels due to indirectness) that text message‐based interventions may have increased vaping cessation rates com… | matched | input-limited (certainty_reasons) |
| T8 | Synthesis of results | One of the studies investigating text message‐based interventions did report on SAEs; however, zero events were reported in both study arms (508 participants; l… | matched | input-limited (certainty_reasons) |
| T9 | Authors' conclusions | Authors' conclusionsThere is low‐certainty evidence that text message‐based interventions designed to help people stop nicotine vaping may help more youth and y… | matched | — |
| T10 | Authors' conclusions | Data exploring the effectiveness of combination NRT, cytisine, and nicotine/vaping behaviour reduction are inconclusive due to risk of bias and imprecision.Most… | matched | input-limited (certainty_reasons) |
| T11 | Authors' conclusions | No studies assessed the change in combustible tobacco smoking, including relapse to or uptake of tobacco smoking, at six‐month follow‐up or longer. | matched | — |
| T12 | Authors' conclusions | Their incorporation into the evidence base and the continued identification of new studies is imperative to inform clinical and policy guidance on the best ways… | matched | input-limited (study_status_or_search) |
| T13 | Authors' conclusions | Therefore, we will continue to update this review as a living systematic review by running searches monthly and updating the review when relevant new evidence t… | matched | input-limited (review_process) |
Two reviews have been searched so far (oral nicotine pouches; riboflavin for blood pressure), with strategies written from the protocol alone. Every study the published reviews included and that is indexed in an accessible source was retrieved, and the PRISMA-S record is complete. For the first review the retrieved records plus every indexed report of the review's included and excluded studies were screened by title and abstract in two independent passes: every retrieved included report went to full text, the passes agreed at kappa 0.97 on 1,351 records, and four conflicts were adjudicated. Full-text screening and the four remaining reviews are still to come.
| Review | Run | Unique records | V1a recall, accessible | V1b recall, all | V1c documentation |
|---|---|---|---|---|---|
| CD016220 | it15-cd016220-SIs | — | 4 / 4 | 4 / 4 | 1 / 1 |
| CD015464 | it15-cd015464-SIs | — | 4 / 4 | 4 / 4 | 1 / 1 |
Extraction (two independent passes and a reconciliation, from open-access full texts, registry results and abstracts): characteristics were populated for every study; the analysis structure matched the published review on three of four reviews above the 85% threshold; exact data rows reached the 90% threshold on two reviews. Rows the published review built by imputing missing outcomes, rows whose numbers occur in no open source, and rows that match with the arms exchanged are reported separately under addendum 5 and listed in the record.
Certainty: the skill's GRADE suggestions, finalised by a judging context, matched the authors' level exactly in 61% and within one level in 95% of 41 Summary of findings rows (thresholds met twice). Writing: every number traced, every Summary of findings row present, 48 of 51 published abstract claims matched with none contradicted (threshold met once), and 4 twelve-word overlaps with the published text where the threshold is 0.
| Review | Run | Study labels | Data rows exact | Analysis structure | Characteristics | Risk-of-bias domains |
|---|---|---|---|---|---|---|
| CD016058 | it04-cd016058-SIx | 5 / 9 | 31 / 38 | 39 / 43 | 44 / 44 | 36 / 54 |
| CD016220 | it05-cd016220-SIx | 4 / 4 | 5 / 6 | 5 / 8 | 16 / 16 | 8 / 16 |
| CD015464 | it10-cd015464-SIx | 3 / 4 | — | — | 28 / 28 | 0 / 4 |
| CD015033 | it12-cd015033-SIx-r3 | 11 / 11 | 3 / 23 | 3 / 15 | 48 / 48 | 6 / 11 |
| CD016278 | it13-cd016278-SIx | 6 / 6 | 19 / 21 | 13 / 15 | 24 / 24 | 22 / 36 |
| CD015746 | it13-cd015746-SIx | 14 / 19 | 25 / 27 | 18 / 21 | 84 / 84 | 10 / 15 |
| Review | Run | Exact level | Within one level | SoF rows present |
|---|---|---|---|---|
| CD016058 | it09-cd016058-SIc | 5 / 8 | 8 / 8 | 8 / 8 |
| CD016220 | it09-cd016220-SIc | 4 / 6 | 6 / 6 | 6 / 6 |
| CD015464 | it09-cd015464-SIc | 2 / 3 | 3 / 3 | 3 / 3 |
| CD015033 | it09-cd015033-SIc | 4 / 6 | 6 / 6 | 6 / 6 |
| CD016278 | it09-cd016278-SIc | 5 / 8 | 8 / 8 | 8 / 8 |
| CD015746 | it09-cd015746-SIc | 5 / 10 | 8 / 10 | 10 / 10 |
| Review | Run | Numbers traced | SoF rows present | Claims matched (contradicted) | Verbatim overlaps | Integrity |
|---|---|---|---|---|---|---|
| CD016058 | it14-cd016058-SIw | 386 / 386 | 8 / 8 | 13 / 13 (0 contradicted) | 2 | pass |
| CD016220 | it14-cd016220-SIw | 457 / 457 | 6 / 6 | 10 / 11 (0 contradicted) | 2 | pass |
| CD015464 | it14-cd015464-SIw | 173 / 173 | 3 / 3 | 5 / 5 (0 contradicted) | 0 | pass |
| CD015033 | it14-cd015033-SIw | 382 / 382 | 6 / 6 | 10 / 11 (0 contradicted) | 0 | pass |
| CD016278 | it14-cd016278-SIw | 263 / 263 | 8 / 8 | 4 / 5 (0 contradicted) | 0 | pass |
| CD015746 | it14-cd015746-SIw | 279 / 279 | 10 / 10 | 6 / 6 (0 contradicted) | 0 | pass |
The statistical engines, the data-package reproduction and the 28-review benchmark of edition 1.0.0 are unchanged in substance; the engine changes since 1.0.0 are RevMan rounding of displayed effects, risk-of-bias downgrade suggestions from weight shares, inconsistency suggested from I² ≥ 40%, a pooled comparator risk by default, Q-based I² for common-effect analyses and unique study counts in result brackets. The 1.0.0 acceptance suite still passes, and the full-pipeline render check was re-run with this edition's scripts on all 28 reproduced reviews.
The 1.0.0 evidence (showcase on Electronic cigarettes for smoking cessation, the tier-A benchmark, the coverage audit, GRADE calibration) stays published at its own address and describes edition 1.0.0.
| Review (PMCID) | Analyses | Route | GRADE/SoF | Report | Package tables | Status |
|---|---|---|---|---|---|---|
| PMC11481097 | 36 | ok | ok | ok | ok | pass |
| PMC11514540 | 39 | ok | ok | ok | ok | pass |
| PMC11559622 | 18 | ok | ok | ok | ok | pass |
| PMC11706636 | 44 | ok | ok | ok | ok | pass |
| PMC12002006 | 756 | ok | ok | ok | ok | pass |
| PMC12043208 | 19 | ok | ok | ok | ok | pass |
| PMC12053466 | 24 | ok | ok | ok | ok | pass |
| PMC12135146 | 13 | ok | ok | ok | ok | pass |
| PMC12151630 | 66 | ok | ok | ok | ok | pass |
| PMC12178777 | 25 | ok | ok | ok | ok | pass |
| PMC12269361 | 25 | ok | ok | ok | ok | pass |
| PMC12365945 | 83 | ok | ok | ok | ok | pass |
| PMC12540017 | 49 | ok | ok | ok | ok | pass |
| PMC12541901 | 9 | ok | ok | ok | ok | pass |
| PMC12549188 | 67 | ok | ok | ok | ok | pass |
| PMC12599494 | 123 | ok | ok | ok | ok | pass |
| PMC12645533 | 58 | ok | ok | ok | ok | pass |
| PMC12713157 | 6 | ok | ok | ok | ok | pass |
| PMC13041208 | 10 | ok | ok | ok | ok | pass |
| PMC13082890 | 84 | ok | ok | ok | ok | pass |
| PMC13329900 | 10 | ok | ok | ok | ok | pass |
| PMC13353907 | 11 | ok | ok | ok | ok | pass |
| PMC13370774 | 45 | ok | ok | ok | ok | pass |
| PMC13386461 | 37 | ok | ok | ok | ok | pass |
| PMC13393078 | 13 | ok | ok | ok | ok | pass |
| PMC13506250 | 167 | ok | ok | ok | ok | pass |
| PMC13509011 | 43 | ok | ok | ok | ok | pass |
| PMC13576036 | 62 | ok | ok | ok | ok | pass |
Edition 2.0.0-rc1 contains 117 files and 1,734,639 uncompressed bytes. Both test suites were replayed from a clean extraction of the exact ZIP in an empty directory; the render check ran on the release tree; Codex CLI discovered the extracted package natively. Each fix made during the fifteen iterations has a regression test that fails on the code it corrected (iteration log).
| Check | Result |
|---|---|
| Package | meta-analysis-2.0.0-rc1.zip · 117 files · 1,734,639 bytes unpacked · SHA-256 936632b8e4f4a8af5f05e87c28f2fc925331f2fa261645b3ddff2921a44cc55f |
| Clean-extraction replay: unit and regression tests | 92 tests · OK |
| Clean-extraction replay: 1.0.0 acceptance suite | round clean_extraction_2026-10-15 · 43 tests · 0 failures · 0 blocked · 12 / 12 contract items pass |
| Render check | 28 / 28 reproduced reviews rendered end to end |
| Native host: Codex CLI | codex-cli 0.162.0 · exit 0 · discovery of the extracted ZIP, no script executed |
| Native host: Claude Code | 2.1.295 · headless exit 1 (OAuth session expired); every blind run of this edition was executed in Claude Code sessions |
117 files · 1694 KiB unpacked · standard Agent Skills layout
Package delivery is not available yet. These instructions describe installation once you have the ZIP. The compatibility table records the actual test date, environment and scope for each agent.
Installation overview. Follow the steps for your agent and check the compatibility table for the tested scope.
Preview listing only: checkout and package delivery are not enabled. Release candidate: see the benchmark record for what has and has not been met.
Preview — package delivery is not available yet
The steps below describe installation once you have the ZIP. Checkout and downloads are not enabled.
R 4.3 or newer. JAGS, rjags and gemtc are optional (Bayesian network meta-analysis); pdftotext is optional (open-access PDFs). The search and retrieval scripts need network access to NCBI, Europe PMC, ClinicalTrials.gov and OpenAlex.
Rscript -e 'install.packages(c("meta","metafor","netmeta","bayesmeta","jsonlite","ggplot2"))'
# optional: sudo apt-get install jags poppler-utils && Rscript -e 'install.packages(c("rjags","gemtc"))'The guard refuses an existing destination so local edits are not overwritten.
test ! -e ~/.codex/skills/meta-analysis && mkdir -p ~/.codex/skills/meta-analysis
unzip meta-analysis-2.0.0-rc1.zip -d ~/.codex/skills/meta-analysistests/test_v2.py holds the regression tests of the review stages (no network needed); run_all.py writes PASS / FAIL / BLOCKED per 1.0.0 contract item; BLOCKED means an R package is missing and is never reported as a pass.
cd ~/.codex/skills/meta-analysis
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_*.py'
PYTHONDONTWRITEBYTECODE=1 python3 tests/run_all.py --round local
python3 scripts/reproduce.py --package fixtures/CD010216_data_package/CD010216-SUP-07-dataPackage.zip --out ../ma-run-CD010216| Agent | Status | Method | Notes |
|---|---|---|---|
| Codex CLI 0.162.0 | Verified2026-10-15 | Native discovery from ~/.codex/skills of the clean extracted 2.0.0-rc1 ZIP (temporary install, removed afterwards) | Linux x86_64. Codex read SKILL.md, named the edition (2.0.0-rc1), summarised the end-to-end scope and named two bundled scripts (ma_validate.py, ma_engine.R); exit 0. Discovery only; the test suites were replayed separately from the same ZIP. |
| Claude Code 2.1.295 | Partial2026-10-15 | Every blind replication run of this edition was executed in Claude Code sessions from the workshop copy; discovery of the packaged ZIP by a fresh headless session could not be recorded | The headless `claude -p` attempt on the temporary install failed to authenticate (OAuth session expired), as it did for 1.0.0. The staged review workflow (independent screening, extraction and judging contexts) was run through Claude Code sub-agents. |
| Antigravity / Gemini Not tested | Untested | Explicit file-reading pointer | This edition was not invoked in these hosts. |
| claude.ai (web), OpenClaw, other agents and operating systems Not tested | Untested | Standard SKILL.md layout; needs a host that can run Python, R and reach the public literature services | No Windows, macOS, web-sandbox, external-importer or buyer-delivery claim is made. The web sandbox cannot run the R engines or the network scripts. |
Organization-use commercial license for the original Nexitia skill material (scripts, references, templates, tests, benchmark documents). The three bundled Cochrane data packages remain CC BY 4.0 works of their authors; transcribed trial data are facts. R, JAGS and the R packages are installed separately under their own licenses and are not bundled; the search and retrieval scripts call public online services under those services' terms. Public evidence and figures are CC BY 4.0. Edition 1.0.0 (2026-10-13) was the first distribution under the same license and is preserved.
The displayed price does not change the license. Terms apply to the identified edition; earlier grants and third-party rights are preserved.
Original owner-commissioned workflow written in October 2026. Edition 1.0.0 (13 October) qualified the engines against 28 open-access Cochrane reviews; edition 2.0.0-rc1 (15 October) adds the review stages and their blind replication benchmark. The package contains the skill text, Python and R scripts, templates, tests, benchmark documents and three CC BY 4.0 Cochrane data packages used as fixtures; no dependency code is bundled.
SHA-256 of meta-analysis-2.0.0-rc1.zip — verify your download against this hash:
936632b8e4f4a8af5f05e87c28f2fc925331f2fa261645b3ddff2921a44cc55f