
Validation and replication
Source:vignettes/validation-and-replication.Rmd
validation-and-replication.RmddisappR is checked in three ways:
-
Simulations with known answers. An independent
implementation of the app’s models and decomposition was run on 50
development scenarios (4,720 simulated datasets) and on ten held-out
scenarios with expectations fixed beforehand. The full report is
installed with the package:
system.file("validation", "VALIDATION.md", package = "disappR"). - Published examples. Each bundled example ships with the settings that match its paper; the comparison with the published estimates is below.
-
Held-out real data.
inst/validation/heldout/holds a protocol and a runner for published datasets that were not used to build the app (Rscript tests/scripts/heldout_validation.R).
Published examples
Each example ships with the column mapping and model settings that correspond to the published analysis. The table below compares what the paper reported with what the app returns on the bundled data using those defaults. “Qualitative” means the direction and the conclusion match but the models are not identical.
| Example | What the paper reports | What the app returns | Verdict |
|---|---|---|---|
| Bichet 2022 — tern haemagglutination | Within-individual +0.089/yr, among-individual +0.002/yr; the two do not differ, so no selective (dis)appearance | Model 3: within +0.091/yr, among +0.002/yr; Model 4 ranks first but Model 1 is only 2.6 AIC behind | Quantitative match |
| Bichet 2022 — tern haptoglobin | No age effect, no selective (dis)appearance | Model 1 has the lowest AIC; no model is lower by more than ~2 | Qualitative match (coefficients differ; the archived haptoglobin scale is ambiguous) |
| Wynn 2025 — tern navigational efficiency | Among-individual −0.870°/yr, within-individual +0.851; season (spring) +9.74; variances 63.3 / 6.3 / 1139.9 | Among −0.78, within +0.81 (varies +0.81 to +0.90 across optimiser starts); spring +4.83; variances 62.5 / 3.4 / 1429 | Qualitative match on the signature (negative among, positive within); the season contrast is about half the published value |
| Sanghvi 2022 — beetle male weight | Apparent late-life weight increase attributed to selective disappearance of lighter males; lifespan effect P < 0.001 | Model 1 shows the same shape (age −0.604, age² +0.113); lifespan +0.433 in Model 6; lowest AIC for Model 5 (Model 6 +57, Model 1 +468) | Qualitative match, consistent with the paper’s interpretation |
| Sanghvi 2022 — beetle female fecundity | Negative binomial; hot developmental and hot adult temperatures each accelerate the decline in fecundity; lifespan included for selective disappearance | Model 4 first, Model 6 +6.4, Model 1 +610 under a Gaussian approximation | Not verified: the published coefficients need the negative-binomial fit (glmmTMB), which the offline check could not run |
| Allain 2023 — chipmunk weaning probability | Binomial GLMM; full model with AFR beats age + age² + lifespan by 23.6 AICc | Binomial GLMM: AFR models take the top two places; models without AFR are 19.5 to 25.8 AICc behind | Quantitative match on the AFR contrast; the season × age terms are unstable in the subset with complete lifespan |
| Allain 2023 — chipmunk litter size | Poisson GLMM; AFR model beats age + age² + lifespan by 25.3 AICc | Poisson GLMM: AFR models first; non-AFR models 11.5 to 17.9 AICc behind | Qualitative match, weaker than published |
| Warner 2016 — turtle egg mass | Egg mass rises with reproductive age (quadratic, r² = 0.68, P < 0.001) | Age +0.566, age² −0.145: rising and decelerating | Qualitative match |
| Warner 2016 — turtle clutch size | No significant age effect (r² = 0.12, P = 0.18) | Model 1 first; age −0.201, age² +0.018 | Qualitative match (no age signal), though the published trend is slightly positive |
| Warner 2016 — turtle hatchlings alive | Fitness declines at old ages | Model 4 first but Model 1 only 2.3 AIC behind once the year random intercept is included | Weak: the app finds no decisive support either way |
| Bouwhuis 2009 — great tit recruits | Cross-sectional peak 3.45 yr; individual-level peak 2.80 yr; ALR +0.04 (selective disappearance); ALR × age not significant; post-peak slope −0.11 | Peak 3.48 → 2.80 once ALR is fitted; ALR +0.062; Model 2 first (Model 4 +3.4, Model 1 +7.5); post-peak slope −0.105 | Quantitative match |
| Bouwhuis 2009 — clutch / brood / fledglings | +0.52/−0.06, +0.62/−0.10, +0.63/−0.11; no ALR effect on any of the three | +0.499/−0.058, +0.594/−0.096, +0.637/−0.109; ALR −0.035, −0.026, +0.034 (all negligible); Models 1, 2 and 5 within 4 AIC | Quantitative match |
| Sanghvi 2025 — fly fecundity | Zero-inflated negative binomial; lowest AIC for Model 4 (ΔAIC for Models 1, 2, 3, 5 = 38.4, 24.1, 28.0, 2.7) | Not reproducible outside R: the zero-inflated fit needs glmmTMB | Not verified here |
| McKenna-Ell 2023 — sheep offspring birth weight | Age −0.061, age at last observation +0.016, capture age +0.112, sex +0.106, twin −0.826; early-life terms and their interactions with age n.s.; variances 0.066 / 0.057 / 0.004 / 0.115 | Model 2 with the example’s covariates, unstandardised (offline ML refit): every slope identical to three decimals (bred as a yearling +0.054); variances 0.065 / 0.055 / 0.004 / 0.114 (ML; the paper’s are REML). Krish confirmed the match in the app | Quantitative match |
| McKenna-Ell 2023 — breeding probability and offspring survival | Breeding: age −0.487, age at last observation +0.188, age × bred as a yearling −0.210. Survival: age −0.232, age at last observation +0.089, early-life recruitment +0.869 | Model 2, binomial (glmmTMB), same terms and random effects. An
independent Python Laplace refit of this specification reproduces every
coefficient and variance to three decimals;
tests/smoke_test.R compares the app’s own glmmTMB fit |
Independent reproduction; app fit to confirm in R |
| Szejner-Sigal 2025 — bee locomotor activity | Negative parabola with age; peak at the day-28 occasion in females and day 21 in males; early-life activity only weakly linked to lifespan | Model 1 + lifespan (offline ML refit): peaks at 26.1 days (females) and 17.3 days (males), whose nearest weekly occasions are days 28 and 21; lifespan n.s. in both sexes; a housing-box random intercept does not improve AIC. Letting lifespan interact with age (Model 6) lowers AIC by about 11 (females) and 14 (males) | Qualitative match, peaks consistent with the reported occasions |
How these checks were made
The comparisons were computed outside R with a re-implementation of the app’s data pipeline and a maximum likelihood mixed model whose deviance was verified against a dense-matrix likelihood, plus Gauss-Hermite binomial and Poisson mixed models verified against brute-force quadrature. They predict what the app reports; they are not a substitute for running the app itself.
Three differences between these checks and the published models are worth stating:
- the published great tit and chipmunk models use several crossed random effects, whereas these checks fit one random intercept and absorb the rest as fixed factors;
- the beetle fecundity and fly examples need count families that the offline check could not fit;
- the tern deflection series is derived here from raw positions, while
the paper computed it after FLightR particle filtering (see
inst/app/data/PROVENANCE.md).
Where the app and the paper disagree, the disagreement is reported above rather than reconciled.
Note on the great tit sample
The bundled file is the paper’s full sample of 7,341 breeding attempts by 4,935 females. An earlier build used a 7,126-row subset that excluded 215 attempts with no lay date; refitting on the full sample moves the individual-level peak from 2.85 to 2.80, exactly the published value, and brings the three component traits closer to the published coefficients as well.