Before you trust the number.

A great many models are invalid because the data pipeline leaked. The model is fine; the dataset already contained the answer. LeakGuard is a Python library that looks for that before anyone quotes the metric.

You describe what your columns mean — which one is the entity, when the prediction had to be made, when each feature became knowable, when the label resolved. LeakGuard turns that description into 32 checks, and tells you which ones it could not run.

32 detectors 611 tests 9 synthetic failure cases MIT licence

$ leakguard audit data.csv --contract contract.yaml

[ERROR] LEK004  'collections_contact_count' is measured after
                the outcome is known for 368 row(s)
     columns: collections_contact_count, prediction_time

      This feature's value is recorded after the label was
      determined, so it is plausibly a consequence of the
      outcome rather than a predictor of it — a collections
      note written after a default, a discharge code recorded
      after the event. Models trained on it score near-perfectly
      offline and are useless in production, where the field is
      empty at scoring time.

      Action: Remove 'collections_contact_count' from the feature
      set. If an earlier snapshot of the same quantity exists,
      join it as of the decision time instead.

      evidence:
        n_rows_after_label_time: 368
        fraction_of_rows: 0.306667
        max_lateness_after_label: 9 days 00:00:00
        label_time_source: decision_time + 14D
        per_split: {"train": 250, "test": 118}

exit status 1

Real output, unedited, from the bundled post_outcome_measurement failure case.

Leakage is a property of the pipeline, not the model.

Every one of these produces an excellent offline metric and a worthless production model. None of them is visible in a learning curve, a confusion matrix, or a cross-validation score — because the validation set is contaminated too.

The tooling that would catch them is not a better model. It is a description of what the columns mean, checked against the table.

Random splitCustomers contribute five rows each and the split was drawn per row, so the model is scored on customers it memorised.
Post-outcome fieldA collections note written after the default. Present in training, empty at scoring time.
Global scalingfit_transform(X) before train_test_split. The held-out rows contributed their own mean.
Label horizonA time-ordered split with no embargo. The last training labels resolved using events from inside the test period.
OrderingPositives loaded last, then a surrogate key assigned. The row id alone separates the classes.

The declarative object

The contract is where the power comes from.

LeakGuard cannot infer semantics from a table. Almost every leakage check is only possible because a contract states which timestamp is the decision point and when each feature became knowable.

The highest-value declaration is availability. available_at or available_after turns feature leakage from a statistical guess into a counted fact: two declared timestamps compared, no thresholds involved.

Every field you omit disables the checks that depend on it — and the report says exactly which ones. A dataset audited with a bare DataContract(target="y") will look almost clean and mean almost nothing.

entity: customer_id          # must not straddle a split
timestamp: event_time        # when the event happened
prediction_time: prediction_time
                            # when the score had to exist
target: default
label_window: 30D           # outcome known 30 days later

split:
  column: split
  time_ordered: false       # cross-sectional, by design

features:
  - name: bureau_score
    dtype: float64
    min: 300
    max: 850
    available_after: 0D     # known at decision time
  - name: last_12m_spend
    available_after: 1D     # one-day reporting lag
  - name: collections_calls
    available_at: collections_time
                            # per-row availability
  - name: customer_uuid
    role: identifier        # screened, not modelled
  - name: ingestion_batch
    role: metadata          # carried, never audited

A contract is YAML or a Python object. Both round-trip losslessly.

Two kinds of evidence, and they are not read the same way.

The distinction is enforced by the architecture, not by convention. A heuristic detector cannot emit an ERROR: the registry rejects the declaration at import time, the finding helper rejects it at emission, and the audit runner rejects it again for any detector that tries to build a finding by hand.

Deterministic

The declared data semantics make the violation mechanically identifiable. A timestamp compared with a timestamp, a set intersected with a set, a column compared with the target. If it fires, the stated relationship is present in the data as given.

Max severity: ERROR
Detectors: 22 of 32
Example: LEK003 — the availability timestamp is later than the decision timestamp on 368 rows.

Heuristic

Suspicious evidence exists, but it does not prove leakage. A statistic compared against a threshold. Both false positives and false negatives are expected, and each detector documents its own.

Max severity: WARNING — enforced, never ERROR
Detectors: 10 of 32
Example: LEK007 — this column alone reaches AUC 0.997, which is worth explaining.

The 32 detectors.

leakguard explain LEK004 prints each one’s method plus its documented false positives and false negatives. Severity below is the maximum a detector may emit; the actual severity of a finding depends on the evidence.

ID	Max severity	Basis	Detects
Data quality · QLT001–QLT009 · 9 detectors Not leakage — but every leakage check is only as trustworthy as the columns it reads.
QLT001	ERROR	deterministic	Contract columns are present in the dataset
QLT002	ERROR	deterministic	Column dtypes match the contract
QLT003	ERROR	deterministic	Null rates per column, and contractual missingness ceilings
QLT004	WARNING	deterministic	Exactly duplicated rows
QLT005	WARNING	deterministic	Duplicated contract keys (entity, or entity+timestamp)
QLT006	ERROR	deterministic	Values outside a declared range or allowed set
QLT007	ERROR	deterministic	Values that cannot be right regardless of domain
QLT008	WARNING	deterministic	Columns with no variation (globally or within a split)
QLT009	WARNING	deterministic	Object columns holding more than one Python type
Split integrity · SPL001–SPL006 · 6 detectors Is the train/evaluation boundary real?
SPL001	ERROR	deterministic	The same entity appears in more than one split
SPL002	ERROR	deterministic	Identical feature vectors appear in two splits
SPL003	WARNING	heuristic	Near-identical rows appear in two splits (heuristic hook)
SPL004	ERROR	deterministic	Split time ranges overlap
SPL005	ERROR	deterministic	Evaluation data predates training data
SPL006	ERROR	deterministic	Split sizes, roles and target balance
Leakage · LEK001–LEK010 · 10 detectors Does the training data contain information the model will not have?
LEK001	ERROR	deterministic	A feature is a deterministic function of the target
LEK002	ERROR	deterministic	A datetime feature holds times after the decision time
LEK003	ERROR	deterministic	A feature becomes knowable only after the decision time
LEK004	ERROR	deterministic	A feature is measured after the outcome is known
LEK005	ERROR	deterministic	Label windows cross the split boundary or overlap each other
LEK006	WARNING	heuristic	A column appears to have been scaled using the whole dataset
LEK007	WARNING	heuristic	A single feature predicts the target almost perfectly (heuristic)
LEK008	WARNING	heuristic	An identifier predicts the target (heuristic)
LEK009	WARNING	heuristic	Whether a value is present predicts the target (heuristic)
LEK010	WARNING	heuristic	A feature looks like a target encoding computed over all rows
Distribution shift · SHF001–SHF004 · 4 detectors Do the splits describe the same population?
SHF001	WARNING	heuristic	A numeric feature's distribution differs between splits
SHF002	WARNING	heuristic	A categorical feature's composition differs between splits
SHF003	WARNING	heuristic	A column's null rate differs between splits
SHF004	WARNING	heuristic	The target's base rate differs between splits
Pipeline fitting · PIP001–PIP003 · 3 detectors Was the fitting confined to training rows? Needs a scikit-learn pipeline.
PIP001	ERROR	deterministic	A transformer was fitted on rows outside the training set
PIP002	ERROR	deterministic	A pipeline step was already fitted before the pipeline was fitted
PIP003	ERROR	deterministic	A transformer was fitted on the entire dataset

A check that did not run is a blind spot, not a pass.

Most leakage tools are silent about their own coverage. LeakGuard ends every report with the detectors it could not run and the declaration each one was missing — in the terminal output, in the Markdown, and as skipped_checks in the JSON.

There are four states a reader has to be able to tell apart, and a report that blurs them is worse than none:

The detector passed — it ran and produced no finding.
The detector found evidence — a finding, at ERROR, WARNING or INFO.
The detector could not run — listed under NOT CHECKED, with the missing declaration named.
The detector is heuristic — every finding it makes is labelled, and capped at WARNING.

findings: 2 ERROR   3 WARNING   3 INFO

source            : data.csv
rows x columns    : 1200 x 12
splits            : train=840, test=360
features audited  : 5 (5 with declared availability)
checks run        : 29 of 32   skipped: 3

────────────────────────────────────────────────
NOT CHECKED
────────────────────────────────────────────────
These detectors could not run. Their silence is
not evidence of absence:

  PIP001  fit_on_non_train_rows
          needs a scikit-learn pipeline; run
          audit_pipeline(pipeline, X, y, ...)
  PIP002  prefitted_transformer
  PIP003  fit_scope_full_dataset

  deterministic findings: 7    heuristic: 1

The header count of features with a declared availability rule is the single best indicator of how much of the leakage family was actually exercised.

Validation evidence

Nine datasets, eight broken on purpose.

All nine are built from the same 240-customer, five-month panel, so the only difference between them is the planted defect. Every value comes from numpy with a fixed seed — no real or private data is used anywhere, and a test asserts byte-identical regeneration. The ninth is a deliberately clean control that must produce zero ERRORs: without it, catching the other eight would prove nothing.

Case	Planted defect	Must fire	Caught
clean_baseline	none — the control	nothing may reach ERROR	0 errors
random_split_longitudinal	rows, not customers, assigned to splits	SPL001	yes
future_timestamp_feature	a feature timestamp after the decision time	LEK002	yes
target_copied_into_feature	the label copied into a feature column	LEK001, LEK007	yes
post_outcome_measurement	a field written only after the outcome	LEK003, LEK004, LEK009	yes
global_scaling_before_split	scaling fitted on the whole dataset	LEK006	yes
duplicate_entities_across_splits	training rows re-ingested into test	SPL001, SPL002	yes
label_window_overlap	90-day labels, time-ordered split, no embargo	LEK005	yes
id_proxy_leakage	surrogate key assigned after sorting by label	LEK008	yes

Three further tests pass when LeakGuard misses a real leak — interaction leakage, leakage below the screening threshold, and global mean-imputation. They are pinned deliberately, so that a future change in behaviour breaks the build and forces the documented blind spot to be updated with it.

Prove it, instead of recognising the fingerprint.

LEK006 spots the arithmetic fingerprint of scaler.fit_transform(X) before train_test_split — a column whose mean is exactly zero over the whole dataset but not over the training rows. That is a heuristic, and it misses global imputation, global PCA, and anything else that leaves no trace.

The pipeline guard removes the guesswork. It wraps every transformer, records the identity of the rows each fit actually saw, and compares them with the training index. Wrapping does not change the fitted model — asserted in the tests, on both the predictions and the learned coefficients.

Only the first wrapped step receives your own rows; later steps see transformed output whose identities live in a different space. Those are audited on the invariant Pipeline.fit actually guarantees — every step sees the same rows the same number of times.

from leakguard.integrations.sklearn import audit_pipeline

report = audit_pipeline(pipe, X, y, train_index=train_idx)
report.raise_for_errors()

# fold-local preprocessing → clean
# fit_transform on everything → caught:

[ERROR] PIP003  step 'prep' was fitted on all 400 rows,
                not just the 280 training rows

      This is the textbook global-preprocessing leak: the
      transformer learned its parameters from the evaluation
      data as well as the training data.

      evidence:
        step: prep
        n_rows_seen_at_fit: 400
        n_train_rows: 280
        n_total_rows: 400

Three pipeline detectors — PIP001 fit outside the training rows, PIP002 pre-fitted steps, PIP003 fit on the full dataset.

The statistics are never taken on trust.

Nothing is inlined into a detector. Every statistic lives in one module and is checked against an independent implementation, a closed form, or arithmetic done by hand — and each test is labelled with which kind of evidence it provides, because the three are not equally strong.

Tolerances are tight on purpose. A widening tolerance is a signal that an implementation drifted, so none was loosened to make a test green.

Statistic	Checked against	Result
roc_auc	sklearn roc_auc_score, incl. heavy ties	exact to 1e-12
roc_auc	Mann-Whitney U by hand	exact
normalized_mutual_info	sklearn, geometric normalisation	exact to 1e-12
population_stability_index	two-bin PSI written out by hand	exact to 1e-12
jensen_shannon_distance	scipy jensenshannon, base 2	exact
two_proportion_ztest	z² == chi2_contingency on the 2×2	exact to 1e-10
ks_2samp, spearman, chi2	scipy, delegation asserted identical	exact
pearson	numpy corrcoef; \|r\| = 1 for affine	exact to 1e-12

What it cannot do.

LeakGuard identifies many common leakage patterns. It cannot prove that a dataset is leakage-free. Whether a value was genuinely knowable at prediction time is a fact about your data-generating process, not about the table. A clean report means “no declared rule was violated and no screen tripped” — never “this dataset does not leak”.

The blind spots are documented per detector, and three of them are pinned as passing tests so they cannot quietly disappear from the documentation:

Nothing declared. No available_at means no LEK003 or LEK004. No label_window means no LEK005. No entity means no SPL001. This is the largest source of missed leakage, and the report names every check it disabled.
Univariate screens miss interactions. Two columns that jointly reconstruct the label while each is individually uninformative are invisible.
Threshold-shaped blind spots. A leak that lifts AUC from 0.70 to 0.85 is large, valuable to a model, and far below any usable screening threshold.
Preprocessing with no arithmetic trace. Global imputation, PCA and feature selection leave no fingerprint. Use the pipeline guard instead of hoping.
Semantics it cannot see. A column that holds the wrong thing, an entity id that is not stable, overlap through an undeclared household or session key, a label definition that changed halfway through the history.

Scope is deliberately narrow: tabular only, in-memory pandas, one dataset per audit, scikit-learn as the only pipeline integration. No repair, no dashboard, no production drift monitoring. The shift checks compare declared splits to judge an evaluation number — they are not a monitoring system.

Install & run.

The Python API is primary; the CLI is a thin wrapper over it and exits non-zero on findings, so leakguard audit works directly as a CI step.

Exit codes distinguish the three outcomes: 0 the audit ran and found nothing, 1 it ran and found problems at or above --fail-on, 2 it could not run at all. Machine-readable output goes to stdout with no preamble; diagnostics go to stderr.

Not on PyPI — that name belongs to an unrelated project, so install from the checkout. Requires Python 3.10 or newer; pandas, numpy, scipy and PyYAML are the only core dependencies.

Browse the detectors

Install from a checkout

The [all] extra adds scikit-learn, rich and pyarrow on top of the four core dependencies.
```
python -m venv .venv
.venv/bin/pip install -e ".[all]"
```
Describe your columns

init-contract guesses from column names and says so at the top of the file it writes.

leakguard init-contract data.csv -o contract.yaml

Audit

Exit 1 on any ERROR. Splits that live in separate files use compare instead.

leakguard audit data.csv --contract contract.yaml
leakguard compare train.csv test.csv -c contract.yaml

Gate CI on it

Machine-readable output, with the severity that fails the build named explicitly.
```
leakguard audit data.csv -c contract.yaml \
  --format json --out report.json --fail-on error
```

Understand a finding

Six subcommands in total; zoo writes the synthetic failure corpus to disk.

leakguard explain LEK004
leakguard checks --category leakage
leakguard zoo --write ./datasets