FALSIFYR ATTACK

Claim card

Claim: treatment -> score
Formula: score ~ treatment + age + baseline_score
Estimate: 0.443
p-value: 0.041
Confidence interval: [0.0251, 0.861]

Survival verdict

COLLAPSES 0/100

The survival score is a heuristic summary, not a formal probability that the claim is true.

Smallest kill

Influential row deletion: remove 1 row -> p = 0.069
Rows: 43
Method: ranked

Attack leaderboard

Rank	Family	Attack	Status	p-value	Explanation
1	row_deletion	Influential row deletion	killed	0.069	remove 1 row -> p = 0.069
2	measurement_error	Binary label flip	killed	0.064	50% kill rate after flipping 2.5% of treatment labels
3	missingness	Median imputation	killed	0.109	Median imputation -> p = 0.109
4	missingness	Mean imputation	killed	0.097	Mean imputation -> p = 0.097
5	split	Random train/test split stability	killed	0.075	50% kill rate after holding out 20.0% of rows
6	split	Stratified train/test split stability	killed	0.057	50% kill rate after holding out 20.0% of rows
7	missingness	Mode/explicit-missing imputation	killed	0.097	Mode/explicit-missing imputation -> p = 0.097
8	measurement_error	Outcome noise	killed	0.057	50% kill rate at noise SD = 0.35 x outcome SD
9	standard_error	HC3 robust standard errors	killed	0.051	HC3 p = 0.051
10	placebo	Fake predictor placebo	survived	0.579	0.0% of random fake predictors were at least as strong as original
11	placebo	Treatment-label permutation	survived	0.522	2.5% of placebo labels were at least as strong as original
12	measurement_error	Predictor noise: age	survived	0.041	no 50% kill rate up to noise SD = 0.35 x age predictor SD
13	standard_error	HC2 robust standard errors	survived	0.045	HC2 p = 0.045
14	standard_error	HC1 robust standard errors	survived	0.044	HC1 p = 0.044
15	specification	Bounded specification search	survived	0.039	claim survived 3 bounded specifications; max p = 0.039
16	covariate	Drop age	survived	0.039	drop age -> p = 0.039
17	standard_error	HC0 robust standard errors	survived	0.039	HC0 p = 0.039
18	covariate	Drop baseline_score	survived	0.033	drop baseline_score -> p = 0.033
19	standard_error	Bootstrap uncertainty	survived	<0.001	bootstrap p approx = <0.001

Family sections

Row deletion / influence

1 killed, 0 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
1	row_deletion	Influential row deletion	killed	0.069	remove 1 row -> p = 0.069

Measurement error

2 killed, 1 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
2	measurement_error	Binary label flip	killed	0.064	50% kill rate after flipping 2.5% of treatment labels
8	measurement_error	Outcome noise	killed	0.057	50% kill rate at noise SD = 0.35 x outcome SD
12	measurement_error	Predictor noise: age	survived	0.041	no 50% kill rate up to noise SD = 0.35 x age predictor SD

Missing data

3 killed, 0 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
3	missingness	Median imputation	killed	0.109	Median imputation -> p = 0.109
4	missingness	Mean imputation	killed	0.097	Mean imputation -> p = 0.097
7	missingness	Mode/explicit-missing imputation	killed	0.097	Mode/explicit-missing imputation -> p = 0.097

Split stability

2 killed, 0 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
5	split	Random train/test split stability	killed	0.075	50% kill rate after holding out 20.0% of rows
6	split	Stratified train/test split stability	killed	0.057	50% kill rate after holding out 20.0% of rows

Standard-error / uncertainty

1 killed, 4 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
9	standard_error	HC3 robust standard errors	killed	0.051	HC3 p = 0.051
13	standard_error	HC2 robust standard errors	survived	0.045	HC2 p = 0.045
14	standard_error	HC1 robust standard errors	survived	0.044	HC1 p = 0.044
17	standard_error	HC0 robust standard errors	survived	0.039	HC0 p = 0.039
19	standard_error	Bootstrap uncertainty	survived	<0.001	bootstrap p approx = <0.001

Placebo

0 killed, 2 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
10	placebo	Fake predictor placebo	survived	0.579	0.0% of random fake predictors were at least as strong as original
11	placebo	Treatment-label permutation	survived	0.522	2.5% of placebo labels were at least as strong as original

Specification

0 killed, 1 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
15	specification	Bounded specification search	survived	0.039	claim survived 3 bounded specifications; max p = 0.039

Covariate dependence

0 killed, 2 survived, 0 unavailable.

Rank	Family	Attack	Status	p-value	Explanation
16	covariate	Drop age	survived	0.039	drop age -> p = 0.039
18	covariate	Drop baseline_score	survived	0.033	drop baseline_score -> p = 0.033

Fragility curves

Influential row deletion

Step	Rows removed	Estimate	p-value	Killed	Rows
1	1	0.355	0.069	yes	43

Binary label flip

Perturbation level	Kill rate	p-value
0.01	40.0%	0.043
0.025	63.3%	0.064
0.05	66.7%	0.071
0.1	56.7%	0.071

Random train/test split stability

Perturbation level	Training rows	Holdout rows	Kill rate	Estimate	p-value
0.2	60	14	73.3%	0.433	0.075
0.3	52	22	83.3%	0.416	0.121

Stratified train/test split stability

Perturbation level	Training rows	Holdout rows	Kill rate	Estimate	p-value
0.2	60	14	60.0%	0.467	0.057
0.3	52	22	70.0%	0.416	0.098

Outcome noise

Perturbation level	Kill rate	p-value
0.05	23.3%	0.043
0.1	13.3%	0.044
0.2	36.7%	0.039
0.35	56.7%	0.057

Fake predictor placebo

Replicate	Estimate	p-value	Killed
1	0.0331	0.882	no
2	0.246	0.266	no
3	-0.328	0.131	no
4	-0.304	0.164	no
5	0.279	0.200	no
6	-0.0325	0.883	no
7	-0.251	0.278	no
8	-0.00114	0.996	no
9	-0.0676	0.757	no
10	-0.245	0.260	no
11	-0.00686	0.976	no
12	0.163	0.459	no

Showing first 12 of 80 curve rows.

Treatment-label permutation

Replicate	Estimate	p-value	Killed
1	0.113	0.609	no
2	-0.143	0.523	no
3	0.248	0.262	no
4	0.212	0.331	no
5	-0.059	0.787	no
6	-0.399	0.066	no
7	0.0965	0.662	no
8	-0.167	0.444	no
9	-0.172	0.450	no
10	0.0371	0.866	no
11	-0.287	0.192	no
12	-0.00802	0.971	no

Showing first 12 of 80 curve rows.

Predictor noise: age

Perturbation level	Kill rate	p-value
0.05	0.0%	0.041
0.1	0.0%	0.041
0.2	0.0%	0.041
0.35	0.0%	0.041

Limitations and caveats

A killed claim is fragile under the named attack. It does not prove the original claim is false, and it does not say the perturbed analysis is preferable.

This report summarizes attacks that falsifyr could run from the fitted object and recoverable data. Unsupported or unavailable attacks are listed below so absence of evidence is not mistaken for robustness.

No requested attack returned an unavailable status.

Reproducibility appendix

Call

attack(model = fit, term = "treatment", attacks = c("row_deletion", "standard_error", "covariate_drop", "missingness", "measurement_error", "placebo", "specification", "split"), intensity = "fast", seed = 11)

Model class

Attack settings

seed = 11; intensity = fast; profile = default; alpha = 0.05; kill rule = p_over_alpha; alternative = two.sided

Attack families

row_deletion, standard_error, covariate_drop, missingness, measurement_error, placebo, specification, split

Unknown requested families

Package version

0.1.0

R version

4.4.1

Platform

x86_64-w64-mingw32

Session info

Show session info

R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26200)

Matrix products: default


locale:
[1] C
system code page: 65001

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] falsifyr_0.1.0

loaded via a namespace (and not attached):
 [1] utf8_1.2.4         RColorBrewer_1.1-3 R6_2.6.1           tidyselect_1.2.1
 [5] lattice_0.22-6     farver_2.1.2       magrittr_2.0.3     gtable_0.3.6
 [9] zoo_1.8-14         glue_1.8.0         tibble_3.2.1       pkgconfig_2.0.3
[13] dplyr_1.1.4        generics_0.1.3     lifecycle_1.0.4    ggplot2_4.0.0
[17] cli_3.6.5          S7_0.2.0           fansi_1.0.6        scales_1.4.0
[21] sandwich_3.1-1     grid_4.4.1         vctrs_0.6.5        compiler_4.4.1
[25] lmtest_0.9-40      pillar_1.9.0       rlang_1.1.7

Started

2026-07-10 22:45:10 UTC

Finished

2026-07-10 22:45:11 UTC

Elapsed seconds

0.978