Open Source2026 / 05

falsifyr

An R package that actively tries to break a statistical result, then shows the smallest plausible change that makes the claim fragile.

RtestthatRobustnessRStudio

Source & deliverables

falsifyr attack report with a claim verdict and ranked robustness attacks

11exported functions

65test blocks

8attack families

40users in past day

01 / Context

The question
behind the work.

Statistical work often stops once a coefficient crosses a threshold. Diagnostics exist, but they are scattered across functions and rarely answer the more useful question: what is the smallest reasonable change that makes the claim disappear?

My role

I designed and built the R package, attack engine, S3 methods, plotting and reporting layers, RStudio addin, vignettes, test suite, and CRAN publication workflow.

02 / Implementation

What I built.

Eight attack families test row sensitivity, uncertainty, missingness, measurement error, placebos, specifications, and sample splits before producing a traceable report.

Wrapped eight attack families around lm, glm, and supported hypothesis-test claims: row deletion, uncertainty, covariates, missingness, measurement error, placebos, specifications, and sample splits.
Ranked the smallest kill by severity so an analyst can see the weakest assumption without reading a pile of disconnected diagnostics.
Added survival scoring, leaderboards, static HTML reports, and visual maps that distinguish fragile results from attacks a claim survives.
Stored formula, term, alternative hypothesis, seed, attack settings, package version, R version, and session details with each result.
Built an optional RStudio addin that finds supported model objects and launches the same controlled workflow from the active session.

03 / In detail

Ask what breaks the result

falsifyr starts after a result looks significant. Instead of treating one coefficient or p-value as the end of the analysis, it asks how stable the claim is when the data, specification, uncertainty estimate, and sample are stressed in reasonable ways.

I built the package around a practical question: what is the smallest change that makes this claim fragile? That turns a pile of disconnected diagnostics into a ranked, reproducible investigation an analyst can actually review.

Eight ways to test a claim

The package supports lm, glm, and selected hypothesis-test workflows through row deletion, uncertainty, covariate, missingness, measurement-error, placebo, specification, and sample-split attacks. Each run records the model, focal term, alternative hypothesis, seed, attack settings, package version, R version, and session details.

The output includes a survival score, smallest-kill result, leaderboard, plots, and a static HTML report. The attack stays visible beside the result, so the user can inspect what changed rather than receiving a black-box robustness label.

Built as a real R package

I built the R package, S3 methods, reporting layer, plots, RStudio addin, vignettes, and test suite as one release workflow. The addin finds supported model objects in the active RStudio session and launches the same controlled analysis without forcing a separate environment.

Version 0.1.0 includes 11 exported functions, 65 testthat blocks, and eight attack families. falsifyr is published on CRAN with deterministic checks, explicit dependencies, and traceable results built into the release.

04 / Engineering judgment

The decisions
that shaped it.

Marked a claim as fragile when a defined attack changes the result, with the exact attack and threshold preserved in the report.
Kept the public API small around attack, smallest_kill, attack_leaderboard, score_survival, report, print, and plot workflows.
Reported unsupported attacks and their reasons so every skipped check is visible in the final report.
Kept deterministic attacks reproducible under a fixed seed and separated optional dependencies from the core install path.

Evaluation & results

falsifyr is published on CRAN. Version 0.1.0 contains 11 exported functions, 65 testthat blocks, and eight attack families; it recorded 40 users in the past day. The package has deterministic checks, explicit dependencies, and traceable results built into the release.

05 / Working outputs