Audit a validation for leakage: does outside ground-truth actually enter, or is everyone confirming a result no one independently produced?

Goal

The name is the

Mandela Effect

— a whole population

confidently remembers

something that never independently happened; a leaky validation is the same shape. Walk the 8 patterns below.

mandela

checks one thing: whether a

validation

is independent, or whether the designer, model, and scorer are only confirming each other.

Workflow

Identify the validation (eval / metric / experiment / holdout / "how we'll know"). Name its

components

— what plays model, scorer, designer, dataset.

Ask the

core question

does external ground-truth enter independently ? Test the validation against all 8 patterns below (some apply only to certain components — a human subject, a scorer); report only the ones that fire, each by name. Give the independent-ground-truth fix for each hit. The 8 leakage patterns Show more Installs 1.0K Repository lilmgenius/paperthin GitHub Stars 1.1K First Seen Jun 28, 2026 Security Audits Gen Agent Trust Hub Pass Socket Pass Snyk Pass