Evidence framework for the Chronic Disease Evidence Explorer

• Status: methods specification, v2.0, built 21 October 2026
• Applies to: dataset v2.0 (45 diseases, 124 risk factors, 431 links)
• Published at: https://epidetective.com/chronic-disease-explorer/framework.html. The Markdown source is docs/FRAMEWORK.md in the dataset repository.
• Companion pages: the change log of every link and the reference list
• Sister project: cancer-evidence-explorer (IARC hazard identification)

Readers of v1.0 will find what changed in section 14.

1. The problem this project faces that the cancer project did not

The cancer explorer had an enormous advantage. IARC had already done the adjudication. Group 1, 2A, 2B and 3 is a published, expert-consensus, internationally recognised classification, so the site only had to present it.

Chronic disease has no IARC. No single body says "high sodium intake is a cause of stroke" with the authority of a Monograph. What exists instead is:

• thousands of overlapping meta-analyses of observational cohorts,
• a well-documented history of confidently held associations that reversed when tested properly,
• and several grading systems (WCRF/AICR, GRADE, USPSTF, GBD, umbrella-review classes) that disagree with one another.

Any project that maps chronic disease risk factors has to face this directly or it will mislead. This document sets out how this project does that. Where a body does rule on a risk-outcome pair, the project records the ruling and uses it as one line of evidence (section 10.2).

2. Critique of the first score proposal

The first proposal was:

score = (number of systematic reviews showing a positive association) × (strength of that association, i.e. RR or meta-RR)

The idea is sound and the project kept it: evidence strength has a replication dimension and a magnitude dimension, and both should count. The specific formula has six defects that would make the output indefensible at peer review.

2.1 A plain product makes unlike things look alike. Twenty reviews each reporting RR 1.1 score 22. One review reporting RR 20 scores 20. The two situations differ a great deal, yet the scores are almost the same. Count and magnitude have to be combined on a scale where each levels off.

2.2 Counting systematic reviews counts research attention. Diet and cardiovascular disease have hundreds of meta-analyses. Occupational silica and chronic kidney disease have a handful. The difference reflects funding, fashion and ease of study, and it says little about the strength of causation. Uncorrected, the score would rank well-funded fields above well-established ones.

2.3 Systematic reviews on the same question share their studies. Eight meta-analyses of the same relationship usually pool largely the same cohorts. Counting them as eight confirmations counts the same participants eight times. The literature calls this the overlapping-reviews problem and measures it as the corrected covered area. Replication needs sharply diminishing returns and a discount for overlap.

2.4 RRs can be compared only when the exposure contrast is fixed. "RR 1.2" means little alone. Per 1 SD? Per 50 g/day? Highest against lowest quintile? Ever against never? Extreme-quantile contrasts routinely give RRs two to three times larger than per-unit contrasts for the same underlying relationship. Comparing them without normalisation is the commonest error in ranking risk factors across a field.

2.5 A multiplicative score mishandles protective factors. Physical activity and colon cancer has RR ≈ 0.80. Under the raw formula that scores below a null association. Direction has to be separated from magnitude, using |ln(RR)|, so that RR 0.5 and RR 2.0 register as equally strong in opposite directions.

2.6 The decisive objection: the formula has no term for bias. Observational chronic disease epidemiology is where bias lives, and the history of the field shows it. Applying "many reviews × large RR" to the evidence base as it stood at the time would have graded each of these as strong:

Every one of these would have scored highly on count × RR. Each was overturned by randomised or genetic evidence. A score that ignores triangulation reproduces the field's most famous failures. Confounding by healthy-user behaviour, reverse causation and selection into exposure groups do not average away with more meta-analyses. More meta-analyses of the same biased cohorts make the estimate more precise and no less wrong.

3. The framework: three quantities

Three quantities are reported separately, because they answer different questions: how sure the evidence is of a cause (the grade), how large the effect is (the estimate), and how much replicated evidence supports it (the score).

3.1 Axis 1: the evidence grade

A five-level grade answers one question: is the relationship causal? The levels are harmonised with the established frameworks. The grade is judged for the outcome the estimate measures (section 6).

The one-line definitions in the third column appear on every panel of the page. None of the five names is an IARC evidence category.

Versions before 2.0 used the names Convincing, Probable, Suggestive, Limited and Contested. Convincing became Strong, Probable became Moderate, Limited became Weak and Contested became Conflicting. Suggestive keeps its name. The weights are unchanged.

The Conflicting grade is deliberate. Standard frameworks have no good home for "the cohorts say one thing and the trials say another". That situation is common, and it is the one readers most need flagged. Vitamin D deficiency and coronary heart disease sits here: cohort studies show a link, and the large supplementation trials and Mendelian randomisation find none. Arsenic in drinking water and type 2 diabetes is a second kind. Syntheses disagree with one another: studies in places with highly contaminated water find a clear association, and a later analysis restricted to the range that drinking-water standards cover finds none.

3.2 How a grade is set: the decision table

Version 1.0 left grade assignment to expert judgement and recorded the inputs. Version 2.0 states the rules. A script applies them to the recorded fields of every link, and a curator confirms or overrides the result.

3.2.1 How the table works

• Each rule reads recorded fields. A rule that fails sets a ceiling, the highest grade the link can hold. The table grade is the lowest ceiling.
• Conflicting is tested first (rules C1 to C4).
• The table lowers grades and never raises them. A link where the table finds no recorded reason for its Weak or Suggestive grade becomes "cannot tell" and goes to a curator. Where a field is not yet recorded, the script reads the text of caveat, review_note, mediation and quote as a proxy.
• Each rule carries a confidence. A change that rests only on low-confidence rules becomes "cannot tell".
• An outcome mismatch never caps a grade. It decides where the link sits (section 6), and the grade is judged for the outcome measured.
• Every departure from the table is stored as grade_override with a reason and appears in the per-link change log.

3.2.2 What each grade requires

3.2.3 The rules for the script

3.2.4 Adjudicating bodies

Rules T3, T4 and T9 treat a few bodies as adjudicators because they rule on causality for a named risk-outcome pair. The list is in section 10.2. Each body counts only for a pair it rated, which the field adjudication.pair_match records. Pooled cohort consortia (PSC, ERFC, CHANCES) size an association and issue no causal verdict. Each counts as one observational line.

3.2.5 Who decides: the script, the curator and the override

• The script is binding for the rules that read structured fields (T1 to T4 and C1) when they fire at high confidence. The build lowers the grade to their ceiling unless the link carries a grade_override.
• The rules that read text (T5 to T11 and C2 to C4) propose a grade. A curator confirms it.
• A curator who departs from the table writes grade_override as {grade, reason}. The build audit fails any link whose grade differs from the table without an override. The override and its reason appear in the change log.
• When two adjudicators rate the same pair more than one grade apart (secondhand smoke and coronary heart disease is the standing example: Burden of Proof gives two stars, and the US Surgeon General judged the link causal in 2006), the link becomes "cannot tell". The curator grades it and names both bodies in the reason.
• A null trial or MR beside a clear association gives Suggestive by default. It gives Conflicting only when the trial or MR tested the same exposure on the same outcome and a curator confirms. The data record this as same_exposure and same_outcome on each trial or MR entry. A trial of a surrogate (exercise and falls, cognitive behavioural therapy, periodontal treatment and HbA1c) records same_outcome as false.
• A high I-squared alone caps a single review at Suggestive. It often reflects spread in the size of an effect while every study points the same way.
• The table never raises a grade. A Moderate link that meets every Strong condition waits for a curator review. There is no automatic upgrade.
• A link marked "cannot tell" goes to a curator. The curator adds the evidence that the table names, where it exists, and records the grade and the reason in grade_basis. Where the evidence cannot be found, the grade that the verification pass judged most likely applies, and the basis says so.

3.3 Axis 2: effect magnitude and measure

Meta-relative-risk with 95% CI, always paired with an explicit exposure contrast and a contrast type:

• binary: exposed against unexposed (current smoker against never smoker)
• per-unit: per standardised increment (per 5 kg/m² BMI, per 20 mmHg SBP, per 50 g/day)
• quantile: highest against lowest category (flagged, and not directly comparable to per-unit)
• category: defined clinical strata

Each estimate also carries its measure: RR, HR, OR, SMR or rate ratio. The page shows the measure beside the estimate and gives each measure its own marker on the plot. For common outcomes an odds ratio overstates the risk ratio, and the page flags these links. It applies no conversion, because a conversion needs a baseline risk for each link.

The contrast type and the measure are always shown. Lists sort by grade by default. A score sort is offered only inside one comparison type, with rows grouped by type.

3.4 Axis 3: the evidence strength score (ESS, 0 to 100)

The score is the composite the project was asked for, rebuilt to survive the objections of section 2.

ESS = 100 × M × R × C

M, the magnitude component (levels off, symmetric in direction):

M = |ln(RR)| / (|ln(RR)| + ln 2)

Half of the maximum is reached at RR = 2 (or 0.5). M keeps rising, so RR 20 still outranks RR 3, and a doubling of RR no longer doubles the score. Taking |ln(RR)| gives protective and harmful effects of equal size the same value. Direction is a separate field.

R, the replication component (diminishing returns):

R = 1 − exp(−N_eff / 2)

N_eff is the effective number of independent syntheses that the link cites, computed by the counting rule of section 5. It is never entered by hand. Under the rule N_eff runs from 1.00 to 2.31 in the current data, so R runs from 0.39 to 0.68:

C, the credibility multiplier, is the grade weight of section 3.1: Strong 1.00, Moderate 0.80, Suggestive 0.55, Weak 0.30, Conflicting 0.25.

Triangulation, heterogeneity, publication bias and reverse-causation risk are assessed when the grade is assigned. They are not applied again as separate multipliers, which would count them twice. Each link records the rules it met and the reason (grade_basis), so the grade can be audited.

How the score appears. Each panel shows the three parts, M, R and C, with one line of explanation, in a neutral colour. The score is stored with the link as ess, and its parts as score_parts. Links in the related-outcome layer carry no score (section 6).

3.5 Worked examples

Each row is a link of dataset v2.0. N_eff comes from the counting rule of section 5.

How N_eff was reached in each row:

• Body-mass index and type 2 diabetes. Two syntheses, from 2010 and 2022. The second comes 12 years after the first and counts 12 / (12 + 5) = 0.71. N_eff = 1.71.
• Smoking and coronary heart disease. A pooled analysis from 2015 (CHANCES) and the Burden of Proof study of 2022. The gap is 7 years, so the second counts 7 / 12 = 0.58. N_eff = 1.58.
• Mediterranean diet. Meta-analyses from 2017 and 2019. The gap is 2 years, so the second counts 2 / 7 = 0.29. N_eff = 1.29.
• Low vegetable intake. A meta-analysis from 2017 and the Burden of Proof study of 2022. The gap is 5 years, so the second counts one half. N_eff = 1.50.
• Processed meat. An umbrella review from 2019 and the GBD 2021 risk-factor analysis from 2024 (5 years later, one half). A third source from 2024 is published in the same year as the one before it and counts 0. N_eff = 1.50.
• Sleep apnoea. Meta-analyses from 2013 (two, published in the same year, so the second counts 0) and 2017 (4 years later, 4 / 9 = 0.44). N_eff = 1.44.
• Vitamin D deficiency. One synthesis, from 2012. N_eff = 1.00. The grade is Conflicting because the large trials and the genetic evidence contradict the cohort estimate, so C is 0.25.

Three things stand out. The score gives a large, well-replicated effect (the first row) several times the value of a small one (the fifth). The grade weight pulls a link down when triangulation contradicts it (the last row). The fourth and fifth rows share N_eff 1.50, so R is equal and only M and C separate their scores.

4. What the score leaves out

These points are stated prominently on the site, because misreading the score is the most likely harm.

1. Public health importance. The score measures how strong the evidence is for a cause-and-effect link. Public health importance is a separate question. A strong risk factor affecting 0.1% of people matters less to a population than a weak one affecting everybody. The population attributable fraction answers that question, and it depends on prevalence. Gestational diabetes and type 2 diabetes scores 34.9, for example, and gestational diabetes affects 5% to 9% of pregnancies in US data. Sedentary behaviour and type 2 diabetes scores 3.9, and many adults sit for more than seven hours a day. The second concerns far more people.
2. Personal risk. The estimates are population-average relative risks. An individual's risk depends on baseline risk, dose, duration, age, genetics and competing risks.
3. Comparison across contrast types. Per-unit, binary, quantile and category contrasts measure different things. Each estimate shows its type and its measure. Links of different comparison types are never ranked against each other by score.
4. Related outcomes. A link whose estimate measures a wider group, a stand-in, a composite or the course of an existing disease carries no score (section 6).
5. Currency. The score summarises a curated snapshot. The dataset is versioned and dated, and the explorer has no live update.

5. Counting replication: the review-counting rule

5.1 Why a rule

In versions before 2.0 the number of reviews behind each link was entered by hand, and no rule said which sources count or how overlap is handled. In 102 of 431 links the hand count was larger than the number of sources the link cites, so the reader could not see the reviews behind it. The hand-entered N_eff equalled the review count minus one, with a floor of 1, in 422 of 431 links, so it recorded no judgement about overlap. Version 2.0 computes both numbers from the sources a link cites.

5.2 The rule

The replication term uses only the sources a link cites. A cited source counts as a synthesis if it is a meta-analysis, an umbrella review, a pooled analysis of individual-participant data from a consortium, or an expert evidence review (GBD Burden of Proof, a Lancet Commission, a WHO review). Single trials, MR studies, cohorts, case-control studies, narrative reviews and methods papers do not count here. They shape the grade through the triangulation fields. A corrigendum never counts.

Syntheses overlap, because they pool many of the same studies. The count is discounted in three steps that need only recorded metadata:

1. One programme or one review counts once. Syntheses from one programme (PSC, ERFC, IPD-Work, ESCAPE, CKD-PC, GBD with Burden of Proof, the Lancet dementia Commission, CTT, BPLTTC) and versions of one review form a single group. Syntheses by the same first author within five years also merge, once the source records carry first_author.
2. An umbrella review counts once with the meta-analyses it covers. An umbrella review forms one group together with every meta-analysis on the same link from its year or earlier.
3. Later groups count less. Order the remaining groups by year (groups from the same year by source id). The oldest counts 1. Each later group counts gap / (gap + 5), where gap is the number of years since the group before it. A group from the same year counts 0, one from 5 years later counts 0.5, and one from 10 years later counts 0.67.

N_eff is the sum of the weights, stored to two decimals. The panel lists every synthesis counted with its group and weight, so a reader can check any link by hand.

SYNTH = {meta, umbrella, pooled, synthesis}
H     = 5                      # years of gap at which a later synthesis counts one half

count(link):
    S = [s in link.sources where s.type in SYNTH
         and s.title does not start with Corrigendum, Erratum, Author Correction or Correction to
         and (s is not a synthesis of trials or MR studies, or s is link.estimate_source)]
    n_reviews = len(S)                       # never more than the number of cited sources

    if n_reviews == 0:
        return n_reviews = 0, n_eff = 1, flag "rests on primary studies"

    # step 1: same programme or same review -> one group
    merge(a, b) if family(a) == family(b) or normalised_title(a) == normalised_title(b)
    # once first_author is recorded:
    merge(a, b) if first_author(a) == first_author(b) and |a.year - b.year| <= 5

    # step 2: an umbrella review covers the meta-analyses before it
    merge(u, m) for umbrella u and meta-analysis m on the link with m.year <= u.year

    # step 3: year discount between groups
    groups = merged groups, sorted by (latest member year, first member id)
    n_eff  = 1                               # the oldest group counts in full
    for i = 1 .. len(groups) - 1:
        gap   = groups[i].year - groups[i-1].year
        n_eff = n_eff + gap / (gap + H)
    return n_reviews, round(n_eff, 2)

Three links of the dataset show the steps.

• High blood pressure and coronary heart disease cites the Prospective Studies Collaboration analysis of 2002 and a pooled analysis of 2013 built on the same collaboration's data. They form one group (step 1). A meta-analysis of 2013 from another group comes in the same year, so it counts 0, and the Burden of Proof study of 2022 counts 9 / (9 + 5) = 0.64. N_eff = 1.64.
• Sugar-sweetened drinks and type 2 diabetes cites an umbrella review of 2019 and a meta-analysis of 2015. The umbrella review covers it (step 2), so both form one group and N_eff = 1.00.
• Processed meat and type 2 diabetes shows the year step (step 3), including a source from the same year as the one before it, which counts 0. Section 3.5 gives the sum.

5.3 Choices behind the rule

• Expert evidence reviews count. GBD Burden of Proof, Lancet Commission and WHO reviews each run their own literature search. All papers from one programme count once.
• Trials and MR stay in the grade. A trial or an MR study tests causality, and the grade already credits it. Counting it in replication as well would credit the same evidence twice. The exception is a trial or MR synthesis that carries the displayed estimate: CTT for statins with coronary heart disease and with ischaemic stroke, the blood-pressure-lowering trial synthesis for blood pressure treatment, the statin-trial meta-analysis for statins and type 2 diabetes, the influenza-vaccine trial synthesis, and the MR meta-analyses behind the four back pain links, smoking and adult asthma, and high uric acid and coronary heart disease. Its record is typed as a synthesis, and it counts once.
• Links with no synthesis. Some links cite no synthesis. Their estimate rests on a trial, an MR study, a cohort or a case-control study. They receive N_eff 1 and a visible flag, "rests on primary studies". The flag keeps the score defined without implying that the evidence is absent.
• The year step stands in for true overlap. A later review re-pools most of the studies of an earlier one and adds new ones in proportion to the years between them. The weight gap / (gap + 5) levels off in the same way as M. Changing 5 to 3 or 8 changes the results little.
• The test for a second line counts groups. The year discount keeps two groups below N_eff 2, so rule T4 counts independent synthesis groups and other lines of evidence.

5.4 Limits of the rule

• The umbrella step assumes an umbrella review covers every earlier meta-analysis on the same link. Some umbrella reviews cover a wider topic and may omit the cited meta-analysis. The assumption errs towards counting less.
• The rule assumes every cited source is on topic for its link. No mechanical check can confirm this. A topic check is made by a curator whenever a source is attached.
• Some pooled records are multi-cohort papers from one research group and sit close to a single primary study. Each counts as one synthesis.
• In the current data N_eff runs from 1.00 to 2.31, so R runs from 0.39 to 0.68. R separates one synthesis from several, and the grade carries the rest of the evidence.
• A source with no recorded year counts in full, and the build flags it.
• A link counts the reviews it cites. A well-studied relationship that cites few reviews scores lower than the literature would suggest. The cure is to attach more verified reviews, each resolved against PubMed first, and each then counts for itself.

6. Related outcomes

6.1 The question

Each link records what its estimate measures: outcome_match and measured_outcome. The class answers one question: does the estimate measure the disease the link sits under?

A link leaves the counts and the score when its estimate measures:

• a wider group that contains the disease (broader): cardiovascular disease under coronary heart disease, all stroke under ischaemic stroke, asthma at all ages under adult asthma;
• a different measure that stands in for the disease (proxy): wheeze, a symptom score, a urine marker, bleeding;
• a combined outcome that adds other events (composite_other): all-cause death, a blood marker, hypertension;
• recurrence, worsening or complications in people who already have the disease, alone or pooled with new cases (progression).

A link whose estimate measures part of the disease (narrower: a subtype, one site, fatal cases only, a late stage) stays in the counts. Its panel title names the measured outcome. A link whose estimate measures the disease itself is exact.

6.2 Conventions

• Acute or major coronary events count as coronary heart disease.
• Fatal-only estimates are narrower.
• Age, sex or population limits leave the class unchanged.
• A prevalence design is graded by rule T5a and leaves the class unchanged.
• Adult "diabetes" with no stated type counts as type 2.

6.3 What the panel says

6.4 Where related-outcome links sit

Related-outcome links form a separate block under the disease, headed "Related outcomes". The title names the measured outcome. The link keeps a grade for what it measured and shows no score (ess is null). It is left out of counts, sorting and the map's link totals. The field related_outcome is true for broader, proxy, composite_other and progression links and is computed at build.

In dataset v2.0, 356 of the 431 links sit in the counts and 75 sit in the related-outcome layer:

Two choices decide many links.

• Disease names follow what the studies measure. Three diseases were renamed: major depression became Depression, chronic low back pain became Back pain, and osteoporotic fracture became Fracture in adults. The studies behind these links measure depression by diagnosis or symptom-scale caseness, back pain of any duration and fracture at any site. The disease ids are unchanged. The rename keeps 24 links in the counts (11, 6 and 7).
• All-stroke estimates leave the ischaemic stroke counts. Seventeen links under ischaemic stroke measure stroke of every type, and three more measure cardiovascular events. They leave the counts. Re-sourcing a link with an ischaemic-stroke estimate brings it back.

7. Inclusion criteria and the position on Mendelian randomisation

Included:

• Systematic reviews with meta-analysis of prospective observational studies (cohort, nested case-control), published in indexed journals
• Meta-analyses of randomised controlled trials, and pooled trialist collaborations (CTT, BPLTTC)
• Mendelian randomisation studies, including MR meta-analyses and large consortium MR
• Pooled individual-participant-data consortia (for example the Emerging Risk Factors Collaboration and the Prospective Studies Collaboration)
• Authoritative expert syntheses: GBD Burden of Proof, WCRF/AICR Continuous Update Project, IARC Monographs (for shared exposures), USPSTF, Lancet Commissions, WHO reviews

Excluded:

• Narrative reviews, editorials and conference abstracts
• Ecological studies

Kept with a ceiling. Some estimates rest on weaker designs, and the grade table caps them:

• a single primary study (rule T5c) or a single trial, MR analysis or cohort, flagged "rests on primary studies";
• meta-analyses restricted to cross-sectional or case-control designs (rules T5a and T5b).

The record of each link states the design behind the displayed estimate: estimate_design is one of cohort synthesis, pooled IPD, mixed, case-control, cross-sectional, trial synthesis, single trial, MR or single cohort, and estimate_source names the source.

Design decision. Restricting the dataset to systematic reviews is right for estimating an association and incomplete for establishing causation. A trial that overturns a literature must be admissible even though it is one study. The framework therefore admits landmark trials and MR as triangulation evidence. They are labelled as such, and they decide the grade through rules T2, T3 and C1 to C4.

Mendelian randomisation estimates. An MR estimate is an effect per unit of genetic liability. It is a different quantity from a person-level contrast. The framework treats it as follows.

• Where a cited synthesis of prospective observational studies exists, the page displays that synthesis and keeps the MR result for triangulation. A cross-sectional synthesis does not qualify: high body-mass index and sleep apnoea displays the MR estimate because the only observational synthesis cited pools prevalence at a sleep study. One exception stands: high blood uric acid and gout displays the MR estimate of 2016 although an umbrella review of observational studies from 2017 is cited beside it and counts in the replication count. The owner may switch that link to the umbrella estimate.
• Where no observational synthesis is cited, the page keeps the MR estimate and labels it "genetic estimate".
• One MR analysis is one line of evidence, so rule T4 caps it at Moderate. One MR analysis in one cohort is capped at Suggestive by rule T5d.

8. Known limitations

• Curated snapshot with no living-review update mechanism.
• Effect estimates are extracted from published syntheses and inherit their biases.
• Grade assignment combines a script and expert judgement. The rules that read text are proxies, and a curator confirms each change they propose. The inputs to every grade are recorded so that it can be contested.
• Some fields that the rules would use are only partly filled. n_studies is recorded for 21 links and i2 for 6, read from the review each link displays; n_cases is recorded for none. Rules T9 and T10 read the field where it exists and the text of the caveat elsewhere. The fields stay empty in the remaining links until curation fills them.
• Coverage is skewed towards well-studied risk factors and high-income-country populations. A missing link marks a gap in synthesised evidence and leaves the question of harm open.
• Diseases and risk factors interact through mediation and confounding by shared causes. The graph shows marginal pairwise relationships and does not resolve causal pathways. A risk factor that acts entirely through another (adiposity, then blood pressure, then stroke) appears as a direct link to both. Mediation is noted per link where established and is not modelled.
• The review count depends on the reviews a link cites. See section 5.4.

9. Data schema

Each link record carries:

risk, disease, direction, rr, ci_low, ci_high, measure, contrast, contrast_type,
outcome_match, measured_outcome, related_outcome,
grade, grade_basis, grade_override, grade_table,
estimate_source, estimate_design,
triangulation{rct, mr, dose_response},
adjudication[], external_grades[], conflict,
reviews_counted[], n_reviews, n_eff, primary_only,
n_studies, n_cases, i2,
ess, score_parts{M, R, C},
mediation, caveat, quote, sources[], provenance, reviewed, review_note

The table lists the fields that the grade table, the counting rule and the related-outcome layer use.

The build maps the old grade names (Convincing, Probable, Limited, Contested) to the new ones when it loads older records. The grade names stored and shown are the new ones.

10. Provenance policy and the top-grade rules

10.1 Provenance

Part of the first dataset carried named, PubMed-verified citations and part did not. The pipeline records this for every link and displays it. Every link is tagged:

• sourced: at least one named published source is attached, and that reference has been resolved against PubMed (title, journal, year, PMID, DOI). All 431 links of dataset v2.0 carry this tag.
• editorial: the estimate reflects the curator's reading of the general literature, with no single citation attached. Such links are labelled "no named citation" everywhere they appear. No link carries this tag today. Version 1.0 listed 184 of 359 links in this class. Later audits sourced or removed them.

A link that is sourced and cites no synthesis carries the flag "rests on primary studies" (section 5.3).

10.2 The top grade: rules T3 and T4 and the adjudicating bodies

Version 1.0 enforced the top grade with one mechanical rule. Version 2.0 replaces it with two rules from the decision table. A link can hold Strong only if both hold.

• T3, causal corroboration. A randomised or Mendelian-randomisation line supports the link, or an adjudicating body has rated this pair.
• T4, more than one line. Two independent synthesis groups under the counting rule, or one group plus a second line for the same pair from another design (a prospective synthesis, a pooled consortium, a trial synthesis or MR). Case-control studies and sources for a broader outcome do not count as a second line. A trial estimate needs a second trial synthesis, an adjudicator or a supportive MR line for the same pair.

A link that fails either rule is capped at Moderate. This is deliberately conservative. Restoring a grade means attaching the missing citation or evidence.

Adjudicating bodies. These bodies rule on causality for named pairs. Each counts only for a pair it rated.

Three further points apply.

• The Prospective Studies Collaboration, the Emerging Risk Factors Collaboration and CHANCES are not adjudicators. They pool cohorts to size an association and issue no verdict on causality. Each counts as one observational line.
• The pair match is checked for every body. The Blood Pressure Lowering Treatment Trialists' Collaboration reports no atrial fibrillation or kidney outcome, so it does not count for high blood pressure with atrial fibrillation or with chronic kidney disease. CTT reports no diabetes or peripheral artery disease outcome, so statins with type 2 diabetes and LDL with peripheral artery disease no longer cite it: the first cites a meta-analysis of statin trials and an MR study, the second an MR study and a trial.
• Burden of Proof ratings serve a second purpose. A 1-star rating sets Weak, a 2-star rating sets Suggestive and a 3-star rating sets Moderate (rule T11).

11. Citation verification

Fabricated references are the most damaging failure mode for a project like this. No reference is stored as a hand-typed identifier. scripts/verify_sources.py resolves each source against PubMed by title and records the returned PMID, DOI, journal and year.

The first pass exposed why this matters. Of 37 sources:

• 7 failed to resolve at all and had to be found by targeted search;
• 4 resolved to the wrong paper. A title-similarity match pointed "SPRINT" at a 2026 meta-analysis in Annals of Medicine, "Look AHEAD" at the U-TURN trial extension, "Steno-2" at a Brazilian cohort study, and "VITAL" at a JGIM commentary about VITAL;
• 3 more resolved to a commentary, a correction notice and a foreign-language summary in place of the primary article: the Lancet dementia Commission, the secondhand-smoke Burden of Proof paper and the Cochrane salt meta-analysis.

Each of these would have been published as a plausible-looking but wrong citation. All 34 journal sources of that pass now resolve to the correct primary article. Three agency reports (WCRF, IARC, WHO) are not PubMed-indexed and carry direct URLs. The output is in validation/source_verification.json.

Later work extended the check.

• The audit of 18 September 2026 re-checked every citation (519 at the time) and found no fabricated or mismatched identifier. It found a withdrawn Cochrane review as the only source for four links and a retracted paper on the reference page. Both were replaced in versions 1.3 and 1.4.
• Every source attached after the first pass is resolved against PubMed before it is attached.
• Every cited source has a record in data/dataset.json with its PMID, DOI or URL, and the reference page lists each one with a link to the paper.
• Two further checks apply whenever a source is attached: the record gains first_author from PubMed, and a curator confirms that the paper is on topic for the link.

12. Automated audit

scripts/audit.py enforces the framework against the data at the end of every build. It exits with an error on any critical finding and writes its findings to validation/internal_audit.json. The checks, by name:

The audit tests the dataset against its own rules. Matching each estimate to the literature is separate work (section 13).

13. What has been checked and what has not

Method. The author ran the checks with AI-assisted verification. AI models compared each filed estimate with the paper it cites, applied the decision table in a second, independent implementation, and read each proposed grade change against its record. The author reviewed the results and decided every change. The change log lists each change with its reason.

Checked.

• August 2026 literature audit: every estimate compared with the paper it cites, for the exact contrast stated.
• September 2026 audit and its follow-up: citations re-checked, 100 flagged links re-checked against their sources, estimates corrected, unsupported links removed.
• October 2026 review: contrasts rewritten in plain language, estimate types stated, exposure context added.
• Grading reform of October 2026: the decision table was applied by two independent implementations, and a verification pass read each of the 97 proposed grade changes against its record. It accepted 62, rejected 23 and could not settle 12 from the record. A further 16 links that the first implementation left alone were also marked "cannot tell". The accepted changes are in the change log with their reasons.

Not checked.

• No outside expert review of the effect estimates or the grades has been carried out.
• The automated audit tests the dataset against its own rules. Matching each estimate to the literature is the AI-assisted work described above, and its errors can be shared with its sources.
• Where a curation pass could not find the evidence that a "cannot tell" link needed, the grade rests on the best estimate of the verification pass, and the grade basis says so.
• The fields n_studies and i2 are filled for 21 and 6 links. Elsewhere they are empty, so rules T9 and T10 rest partly on caveat text.
• Moderate links that meet every Strong condition have not been reviewed for an upgrade.
• Link-level texts (comparison, caveat and notes) are written in English and Chinese. The other languages show the English text until they are translated.

14. Version history

The framework and the dataset share a version number from 2.0 onwards. Link-level changes are in the change log.

What changed for readers of version 1.0

• Grade names. Convincing, Probable, Limited and Contested became Strong, Moderate, Weak and Conflicting. Each grade has a one-line definition for readers (section 3.1).
• Grade assignment. The decision table (section 3.2) replaces expert judgement. Every grade that departs from the table carries a reason.
• The top grade. The single mechanical rule of version 1.0 became rules T3 and T4 (section 10.2). The Prospective Studies Collaboration, the Emerging Risk Factors Collaboration and CHANCES are no longer adjudicators, and the list of adjudicating bodies is checked pair by pair.
• Counting. The review count and N_eff are computed from the sources a link cites (section 5). The worked examples of section 3.5 use the new N_eff and links of this dataset.
• Outcomes. Links whose estimate measures something other than the named disease sit in a related-outcome layer outside the counts and the score (section 6).
• Estimates. The estimate may come from a trial synthesis or a Mendelian randomisation analysis, with a ceiling on the grade (section 7). Each estimate shows its measure (section 3.3).
• The score. The panel shows its three parts. Lists sort by grade, and the score sorts only inside one comparison type (section 3.3).