Fits birth-death models of species diversification in which the
speciation and extinction rates depend on covariates generated by the
process itself, such as the number of lineages, the age structure of the
lineages and their evolutionary distinctiveness, to reconstructed
phylogenies that contain only extant species. The likelihood is
integrated over the unobserved extinct lineages by importance sampling
on augmented trees, and the parameters are estimated by Monte Carlo
expectation-maximization, by the cross-entropy method, or from a
generalized additive model fitted to a simulated likelihood surface,
following Richter, Haegeman, Etienne and Wit (2020)
Monte Carlo conditional-likelihood inference for endogenous-covariate diversification models on phylogenetic trees. R package, version 1.1.0, submitted to CRAN.
devtools::install_github("franciscorichter/emphasis")
| What | Where |
|---|---|
| Repository | https://github.com/franciscorichter/emphasis: source, CI, docs site, install_github() |
| Docs site | https://franciscorichter.github.io/emphasis/: overview, the four covariates and their biological meaning (covariates), usage, function reference, what is established, estimability, real phylogenies, open issues, glossary, and one page per experiment |
| Simulator | https://franciscorichter.github.io/emphasis/simulator.html: forward simulation of every model and link in the browser; its agreement with simulate_tree() is checked by dev/simulator/agreement.R |
| Issues and bug reports | https://github.com/franciscorichter/emphasis/issues |
| Citation | Richter et al. (2020), A general class of species diversification models for phylogenetic trees, Statistica Neerlandica |
Toolchain: R ≥ 3.5, a C++ compiler, and the packages in DESCRIPTION (Rcpp, RcppParallel for TBB parallelism, nloptr, BH headers, ape, DDD, progress; mgcv for the optional survival GAM).
devtools::load_all() # compiles src/ and loads the package
devtools::document() # regenerates NAMESPACE and man/ from roxygen
devtools::test() # unit tests (tests/testthat)
devtools::check() # full R CMD check
R CMD build . # source tarball
CI runs R CMD check on every push to main and on pull requests through .github/workflows/R-CMD-check.yaml (r-lib/actions): macOS and Windows on R release, Linux on R release, devel and oldrel. The suite and what it checks are under Validation.
R/
simulate.R simulate_tree(): forward / batch / conditional (augmentation); model + link resolution;
.expand_pars/.contract_pars between compact and full parameter layouts
inference.R estimate_rates() dispatcher (mcem / cem / gam), control defaults, compare_models()
emphasis.R .mcem_dynamic_fresh(): thinning-based MCEM
bdi.R conditional (BDI) augmentation sampler + .mcem_bdi() (see "Augmentation samplers")
covariate_path.R lineage_table(), covariate_path()
de.R cross-entropy method (CEM) and importance-sampling utilities
gam.R auto_bounds(), train_GAM(), predict_survival(), likelihood-surface GAM
survival.R survival_mc(), survival_surface(), and estimate_rates(cond = "surface")
pipeline.R emphasis_pipeline(): the bounds → GAM → CEM → MCEM chain, kept for compatibility
diagnostics.R diagnose_mcem/cem/gam + print methods
src/
augment_tree.cpp data augmentation: insert extinct / unsampled lineages (thinning proposal)
E_step.cpp parallel augmentation via TBB, IS weights
M_step.cpp NLopt Subplex maximisation of the weighted log-likelihood
div_tree.cpp forward simulation
loglik.cpp, mcem.cpp, rcpp_*.cpp, rsbplx.cpp, rinit.cpp
inst/include/ header-only model code (model.hpp, augment_tree.hpp, ed_covariate.hpp, general_tree.hpp, …)
man/ generated by roxygen; do not edit by hand
tests/testthat/ unit tests
site/ the docs site's sources and builder: build.py, template.html, style.css, src/*.md,
reference.R + reference/ (Rd fragments), public.py (the public filter)
*.html, covariates/, experiments/, assets/
the docs site as built (committed; GitHub Pages serves them); simulator.html is
hand-written; D.html is a redirect the build writes; self-hosted MathJax;
excluded from the package build
assets/simulator/emphasis-sim.js the simulator page's engine: a port of general_tree.hpp and ed_covariate.hpp
dev/ excluded from the package build
audit/ the as-implemented map of the estimator, its hypotheses, findings and reproduction scripts
simulator/ the check that the simulator page draws from the distribution simulate_tree() does
Exported functions: simulate_tree, estimate_rates, compare_models, covariate_path, lineage_table, auto_bounds, diagnose_mcem, survival_mc, survival_surface; and, kept for compatibility, emphasis_pipeline, train_GAM, predict_survival, diagnose_cem, diagnose_gam.
The package ships no data. Inputs are an ape::phylo object, a simulate_tree() result, or a numeric vector of branching times. A tree that is not ultrametric (one with branching times after the present) is refused with a message. The examples below use simulated trees.
Species diversification is modelled as a stochastic birth-death process whose speciation and extinction rates depend on endogenous covariates, quantities generated by the diversification process itself:
$$ \lambda(s,t) = f!\bigl(\beta_0,; \beta_N N(t) + \beta_D D_s(t) + \beta_{ED},\mathrm{ED}s(t) + \beta_K K_s(t)\bigr), \quad \mu(s,t) = f!\bigl(\gamma_0,; \gamma_N N(t) + \gamma_D D_s(t) + \gamma{ED},\mathrm{ED}_s(t) + \gamma_K K_s(t)\bigr) $$
Two link functions $f$ are available:
| Link | Formula | Properties |
|---|---|---|
| Exponential (default) | $f(\beta_0, \eta) = \exp(\beta_0 + \eta)$ | Positive; unbounded; monotonic; $\beta_0$, $\gamma_0$ are log rates |
| Linear | $f(\beta_0, \eta) = \max(0,; \beta_0 + \eta)$ | Non-smooth at zero; monotonic; a rate driven below zero is clipped at zero |
The link is selected with link = "exponential" (the default) or "linear". The linear link is supported.
Four lineage-level covariates are read on the complete tree, the first from its branching times and the others from its topology:
| Symbol | Name | Definition |
|---|---|---|
| $N(t)$ | Diversity (tempo) | Number of lineages alive at time $t$, a readout of the branching times alone |
| $D_s(t)$ | Age-imbalance | $D_s = I_s - M$: a lineage's isolation time $I_s$ (time since it last speciated) relative to the clade mean $M$; centred, so $\sum_s D_s = 0$ |
| $\mathrm{ED}_s(t)$ | Evolutionary distinctiveness | Fair proportion on the complete tree: $\sum_{b \in \mathrm{anc}(s,t)} \ell_b / n_b$ over the branches from the crown to $s$, each divided by the alive lineages below it. Its last term is the pendant age $I_s$; its sum over lineages is Faith's phylogenetic diversity |
| $K_s(t)$ | Ancestral splits | The number of speciation events on the path from the crown to $s$ at time $t$, hidden splits counted |
$D$. Because $\sum_s D_s = 0$, the age-imbalance is a within-clade signal, uncorrelated with diversity: $N$ measures when the clade diversifies, $D$ the age structure of which lineages do. The mean pendant age $M = P/N$ is used only internally to centre $D$ and is not a user covariate. A lineage's pendant age restarts when it splits. Between events every pendant age and their mean grow at unit rate, so $D_s$ is constant from one event to the next; the simulator, the likelihood's compensator and its event terms all read that value, over the same alive multiset of pendant starts. See the biological meaning of D for what $\beta_D$ tests (age-dependent speciation).
$\mathrm{ED}$. The per-lineage form of phylodiversity dependence. It is defined on the complete tree, so hidden lineages count among a branch's descendants and contribute branches, and the augmentation supplies them; it needs the tree's topology (a phylo, not a branching-time vector). The same routine computes it in the simulator and in the likelihood (inst/include/ed_covariate.hpp). Its cost is quadratic in the augmented tree: the per-segment tables over every alive lineage are built once per tree and cached across the parameter values an M-step tries. Because the augmentation builds the complete clade the current parameters imply, an $\mathrm{ED}$ fit is started from the estimate of the nested model (ed from cr, ned from dd) with the $\mathrm{ED}$ coefficients at zero; from a point far above the net rate the augmented trees grow by orders of magnitude and the E-step's clock runs out.
$\mathrm{ED}^{c}$. $\mathrm{ED}^{c}s(t) = \mathrm{ED}s(t) - \overline{\mathrm{ED}}(t)$, the fair proportion minus its mean over the lineages alive at $t$. Fair-proportion $\mathrm{ED}$ is not centred: its clade mean is Faith's PD over $N$, a clade-level quantity the $N$ term already tracks, so in an ned fit $\beta{ED}$ carries a clade-level component and a within-clade contrast at once. The centred covariate carries only the contrast; it sums to zero over the alive lineages and is constant between events. At $\beta{ED} = \gamma_{ED} = 0$ nedc is dd.
$K$. Constant between events, one more on both daughters at a split, hidden splits counted, so like $D$ and $\mathrm{ED}$ it is defined on the complete tree and needs the topology and the augmentation. Its compensator is the segment length. It is the observable trace of what inherited-rate models (BAMM, ClaDS) carry as a latent variable; over 6,433 lineages on 30 complete dd clades its rank correlation with each of $N$, $D$ and $\mathrm{ED}$ is at most 0.29 in magnitude. The thinning proposal reads it per lineage; the conditional proposal refuses it with a reason, since its mean field does not carry $K$.
Setting coefficients to zero recovers the model classes:
| Shortcut | Covariates | Parameters | Hypothesis |
|---|---|---|---|
"cr" |
none | $(\beta_0, \gamma_0)$ | Constant rates (null) |
"dd" |
$N$ | $(\beta_0, \beta_N, \gamma_0, \gamma_N)$ | Diversity dependence (ecological limits) |
"d" |
$D_s$ | $(\beta_0, \beta_D, \gamma_0, \gamma_D)$ | Age-dependent speciation |
"nd" |
$N + D_s$ | $(\beta_0, \beta_N, \beta_D, \gamma_0, \gamma_N, \gamma_D)$ | Both |
"ed" |
$\mathrm{ED}_s$ | $(\beta_0, \beta_{ED}, \gamma_0, \gamma_{ED})$ | Distinctiveness-dependent diversification |
"ned" |
$N + \mathrm{ED}_s$ | $(\beta_0, \beta_N, \beta_{ED}, \gamma_0, \gamma_N, \gamma_{ED})$ | Distinctiveness over and above diversity |
"edc" |
$\mathrm{ED}^{c}_s$ | $(\beta_0, \beta_{ED}, \gamma_0, \gamma_{ED})$ | The same, on the centred covariate |
"nedc" |
$N + \mathrm{ED}^{c}_s$ | $(\beta_0, \beta_N, \beta_{ED}, \gamma_0, \gamma_N, \gamma_{ED})$ | Within-clade distinctiveness over and above diversity |
"k" |
$K_s$ | $(\beta_0, \beta_K, \gamma_0, \gamma_K)$ | Inherited diversification: a speciation-rich ancestry speciates more (or less) |
"nk" |
$N + K_s$ | $(\beta_0, \beta_N, \beta_K, \gamma_0, \gamma_N, \gamma_K)$ | The same over and above diversity |
Models can also be given as a formula: model = ~ N + D, model = ~ N + ED, model = ~ N + EDc, model = ~ N + K. ("ep" is an alias for "d".) Internally a model is a five-slot vector c(use_N, use_M, use_D, use_ED, use_K), where slot 4 is 1 for the raw covariate and 3 for the centred one, and the full parameter vector has twelve slots, with the two $\mathrm{ED}$ coefficients and then the two $K$ coefficients last, so that eight-slot parameter vectors and three-slot model vectors keep their meaning.
Reconstructed phylogenies record only extant lineages; extinct side branches are unobserved. The marginal likelihood integrates over all possible extinction histories $z$:
$$ L_\theta(y_{\text{obs}}) = \int f_\theta(y_{\text{obs}}, z),dz $$
It is estimated by importance sampling: the observed tree is augmented with latent extinct lineages, and each augmented tree is weighted by $f_\theta / q$.
Two Monte Carlo averages appear in an MCEM fit, and only one of them is unbiased.
The EM objective is an expectation of a log, $Q(\theta)=E[\log f(y,z\mid\theta)]$, and the M-step averages $\log f$ directly (src/M_step.cpp). Averaging logs to estimate a mean of logs is unbiased.
The log-likelihood is not. $\hat\ell=\log\big(\tfrac1N\sum_i w_i\big)$ is a log of a mean, so by Jensen's inequality $E[\hat\ell]\le\log p(y\mid\theta)$: biased down, never up, by an amount that grows as the effective sample falls. AIC inherits it doubled, and it does not cancel between models: the model that is harder to sample is penalised for being harder to sample.
Every fit therefore records details$final_IS$ESS and $gap, and compare_models() reports both and warns when a ranking is not safe against them: when the handicap of a lower-ESS model reaches the AIC gap it is losing by, or when its draw is heavy enough that the handicap cannot be quantified. It also warns when the model it chooses rests on fewer than 10 effective draws. print() on a fit shows the effective sample and, when it matters, how far below the truth the estimate sits.
gap is an indicator of size and not a bound. Against log-normal weights at $N=200$ it overstates the true bias where that bias is negligible (ratio 4.9 at $n_{\text{eff}}=188$, absolute size $3\times10^{-5}$) and understates it where it is not (ratio 0.46 at $n_{\text{eff}}=17$, 0.16 at $n_{\text{eff}}=6$, where the true bias is half a nat). The gap_heavy flag marks $n_{\text{eff}}<N/10$, where the ratio first falls below one half, and a heavy draw is treated as unquantified.
The chosen model is not the lowest AIC. Among nested models the richer one has the lower AIC whenever its two extra parameters raise the log-likelihood by more than 2, which under the null happens on about 13% of trees (the likelihood-ratio statistic is $\chi^2_2$), and more often with several richer models offered beside the null. compare_models() therefore reports chosen: the model with the fewest parameters among those within margin = 2 AIC of the minimum and, among those, the one with the lower AIC, whatever order the fits are passed in; within_margin marks the rows that qualify, and the full ranking is kept.
Three routes to a model. estimate_rates() fits one named model. estimate_rates(model = "all") fits the nested ladder cr, dd, nd, ned, nk (the last two need a phylo), each started from the estimate of the model it nests and boxed by auto_bounds() unless control$bounds names a box per model, and returns the fits with their compare_models() table and its chosen model; control$models restricts the ladder. covariate_path() selects and estimates at once, along one dgLARS path (Augugliaro, Mineo & Wit 2013) over the covariates, on the segment-level Poisson form of the log-linear point process (lineage_table() is that form, one row per lineage and segment), choosing a point by BIC or AIC for speciation and extinction separately. Called on a phylo, it treats the tree as fully observed; the augmented route, with the hidden lineages entering at their importance weights, runs on the draws of an E-step.
What a fit records about itself. A fit carries notes, each stating the number it rests on: a lineage-level covariate on fewer than 50 tips; an effective sample below 10 at the last E-step; turnover above 0.5 at the intercepts; under model = "all" with both nd and ned in the ladder, that the two read the same pendant edge; clock when max_time cut a step short; and mean_field when the surrogate proposal's mean-field iteration was relaxed below the full step (which the default does at every sweep when $\rho < 1$), ran under a damping the user set, or did not converge within its sweep budget; the note names the setting and whether it was the default. print() shows them.
Two proposals $q$ are available, selected with simulate_tree(method = ) for conditional simulation and control = list(sampling = ) for MCEM:
| Sampler | Where | Scope | Property |
|---|---|---|---|
"bdi", the surrogate (default) |
R/bdi.R |
cr, dd and $\mathrm{ED}$ models on the linear and exponential links; $D$ models on request; any $\rho \in (0,1]$ |
Draws from the exact conditional distribution under constant rates (ESS = sample size); under diversity dependence, $D$ or $\mathrm{ED}$ it is a mean-field proposal built on a self-consistent backward-forward iteration, and the weights carry the approximation |
"thinning" (sampling = "dynamic_fresh") |
src/augment_tree.cpp |
Every model and link | Poisson-thinning proposal, C++ with TBB parallelism |
.bdi_supported(model_bin, link, rho) gates both entry points, and .bdi_unsupported_reason() supplies the phrase the fallback message carries. What runs on thinning:
control$sampling names "bdi". Under the surrogate the average lineage has $D = 0$, so the mean field is the dd one and $\beta_D$ enters only the attachment of hidden births, which is exact: each candidate parent is weighted by the model's own speciation rate at that instant. The pre-registered condition for making it the $D$ default is not met.The surrogate's mean field is a Picard iteration, and where its map expands it diverges; control$damping under-relaxes the sweep. By default the step is "auto" at complete sampling ($\rho = 1$), which halves the relaxation whenever the residual grows, and a fixed ⅛ when $\rho < 1$. At the shipped sweep budget (20 sweeps at $\rho = 1$, 200 below) a fixed ⅛ converges more diverging cases below $\rho = 1$ (142 against 131 of 192 units at $\rho = 0.2$) but none of 192 at $\rho = 1$, against 131 for "auto", at 1.4 to 3.3 times the E-step time and with no gain in effective sample (E10). A mean field that converges only under relaxation leaves an effective sample of about 1 either way. Every E-step's mean field is in the trace (mf_converged, mf_delta, mf_sweeps, mf_omega, mf_budget), and fit$notes$mean_field records the relaxation used.
The thinning proposal reads ED. For a model with $\mathrm{ED}$ the sampler draws the missing births at the model's total speciation rate over the lineages alive on the segment, $\Lambda(t) = \sum_s \lambda_s(t)$ with each lineage at its own $\mathrm{ED}s(t)$, and attaches a birth to lineage $p$ with probability proportional to $\lambda_p(t)$; the segment's extinction rate reads the mean $\mathrm{ED}$. The density mirrors it through the ED table (Model::ed_proposal_seg_t, Model::sampling_prob_ed), and at $\beta{ED} = \gamma_{ED} = 0$ it is the mean-field density term for term. The mean-field thinning proposal, $N\lambda(N, M)$ with the ED term at zero and uniform attachment, is reachable through control = list(proposal = "meanfield"); the ED-aware one is the default because it completes where the mean-field one returns zero weight (E3).
The estimator is Monte Carlo EM. At each iteration the E-step draws hidden lineages from the proposal and weights them, and the M-step maximises the weighted complete-data log-likelihood over a box with Rowan's Subplex (NLopt), the first simplex sized at a tenth of the box in every coordinate; the per-lineage tables are built once per E-step and cached across the values the M-step tries.
The box comes from the observed tree alone, by auto_bounds(): a centre at the net rate the tip count and crown age imply, a bisection along each axis toward the point where forward simulation stops producing clades within a tenth and ten times the observed size, each slope axis scaled by its covariate's size on the tree, and a second pass that tightens the box where the augmentation fails at its corners. Its $\lambda$ ceiling has turnover headroom (the implied net rate divided by $1 - 0.95$) and the search probes the constant-net-rate ray, so a high-turnover optimum is inside the box.
Two stopping rules: rel_change (default) stops once the relative step, in units independent of the tree's time scale, falls below 1e-2 on three consecutive iterations; mc_error measures the step against its own Monte Carlo standard error from five batches, grows the draws by half whenever a step is inside the noise, and stops once the draws reach their cap and the step has stayed inside the noise for three iterations, returning the mean of the last three iterates. rel_change compares the step with a fixed tolerance rather than with the noise of the iterate, so on dd fits at 200 draws it can run to its cap; mc_error lands about 2% from a 400-iteration reference (E9). Both are subject to max_iter and max_time, and a fit reports which of the four ended it.
control$max_time bounds the E-step and the M-step: both samplers read the clock between draws and inside a draw, the M-step's optimiser reads the same deadline before every evaluation of its objective and keeps its best point only if it improves on the current iterate, and the fit ends with stop_reason = "time_budget" and a clock note. After the loop one more E-step runs at the returned point, with max_time again, so that the reported log-likelihood, effective sample and gap describe the estimate and a fit returns within about twice max_time.
estimate_rates(method = "cem" | "gam") and emphasis_pipeline() are further estimators (a cross-entropy search, a likelihood surface fitted on a grid); nothing above depends on them. survival_mc(), survival_surface(), train_GAM() and predict_survival() are the survival probabilities of the next section.
Reconstructed trees are conditioned on survival to the present. The conditioned log-likelihood is:
$$ \ell_{\text{cond}}(\theta) = \ell(\theta) - \log P_\theta(\text{survival}) $$
Survival is crown survival: both crown lineages leave a sampled descendant at the present. The likelihood is not conditioned by default (cond = NULL), and neither is the ladder. cond takes one of:
"surface": $\log P_\theta$ estimated by direct simulation on local quadratic surfaces around the iterate, for any model and link, under MCEM. The recommendation for the lineage-level models.cr and dd, with the exact value: under cr on the exponential link one crown lineage leaves a sampled descendant with probability $p = \rho d / (\rho\lambda + (\lambda(1-\rho) - \mu)e^{-dT})$, $d = \lambda - \mu$, and $P_\theta = p^2$; the example in ?estimate_rates codes it. Under dd it is the solution of the master equation on the clade size. Neither function is in the package.survival_surface(), held fixed through the fit.train_GAM(), which simulates clades at Latin-hypercube points of the box and counts a clade that reaches the lineage cap as surviving (auto_bounds(train_surv_gam = TRUE) returns one), or an nnet object. In E43, GAMs trained on 2000, 5000 and 20 000 clades for each of the eight lineage-level models had a 90th-percentile error of $\log P(S)$ of 0.44 to 2.56, against the 0.05 its rule required, and the error did not fall with the training size; the emulator is not accurate enough to condition on. Its accuracy under cr and dd over the whole box is measured in E36.$P_\theta$ does not depend on the observed tree, so one function per model, crown age and $\rho$ serves every fit of a cell.
The direct estimate. survival_mc() draws clades forward to the crown age, clade $i$ from seed seed + i, so that two calls with the same seed draw the same clades at different parameters (common random numbers). A clade that passes the lineage cap counts as surviving, as in train_GAM(). Under $\rho < 1$ a clade that reaches the present with $n_1$ and $n_2$ extant descendants of its two crown lineages contributes $(1-(1-\rho)^{n_1})(1-(1-\rho)^{n_2})$, the probability that sampling each extant tip with probability $\rho$ keeps one on both sides. The estimate is $\log((k + 1/2)/(n + 1))$ over $n$ clades of total contribution $k$, with the delta-method standard error. Against the exact cr value it agrees within its error at $\rho = 1$ and $\rho = 0.7$ (tests/testthat/test-survival-surface.R). At a standard error of 0.01 one estimate took E43 a median of 2 to 29 s of one core under the lineage-level models, and the M-step evaluates cond at every value it tries, which is why "surface" does not call it there.
cond = "surface" runs four steps (R/survival.R, .fit_surface_cond()):
init_pars, to the stopping rule.survival_surface()): direct estimates at $2p + 1$ points, $p$ the coefficients of a full quadratic in the $d$ parameters (31 points for $d = 4$, 57 for $d = 6$), the centre and a Latin hypercube of a box of half-width 0.03 of the fit's box per coordinate; the same clades at every point, as many as give a standard error of 0.02 at the centre; a quadratic fitted by weighted least squares, the tangent plane outside the box, capped at $\log P = 0$.cond, in segments of 25 iterations that carry the draw count forward. The surface is rebuilt at the iterate whenever the iterate leaves the inner half of its box; the run ends when the stopping rule is met inside the inner half, or on max_iter iterations or max_time.init_pars, the nested start (the nested model's estimate, lineage-level coefficients at zero), and the run with the higher conditioned log-likelihood is kept.The reported loglik is the kept run's unconditioned log-likelihood minus a direct $\log P(S)$ at its estimate, at a standard error of 0.01; loglik_var adds that error's square. max_time bounds the four steps together, and the direct estimate at the estimate gets max_time again, as the final E-step does. The fit's conditioning records the surfaces built, the clades simulated and the seconds spent, the direct $\log P(S)$ at the estimate with its standard error, the unconditioned log-likelihood, which start was kept, whether step 4 ran, the last surface, and each run's estimate, stop reason, iterations and trace. In the pilot that chose the method for E46 (100 tips, cr and dd, where the exact conditioned maximum is known), steps 1 to 3 reached a median deficit of 0.0011 nats (worst 0.13) against 0.0074 for MCEM with the exact factor and 0.26 for the unconditioned fit; without step 4 a second, lower maximum on the extinction edge was kept on 4 of 10 cr trees at turnover 0.10. The surfaces cost under 0.3% of a cr/dd fit, 0.4-1.8% under ned and 3-9% under nk.
Its settings are entries of control, read only under cond = "surface":
| Entry | Default | What it sets |
|---|---|---|
surface_radius |
0.03 |
half-width of a surface's box, as a share of the fit's box per coordinate |
surface_se |
0.02 |
standard error of $\log P$ at a surface's centre that the clades per point are set for |
surface_final_se |
0.01 |
standard error of the direct $\log P$ at the estimate |
surface_segment |
25 |
MCEM iterations per segment |
surface_max_lin |
max(1000, 20 * n_tips) |
lineage cap of a simulated clade |
surface_edge |
1e-3 |
share of the box span within which $\gamma_0$ triggers step 4 |
surface_restart |
TRUE |
whether step 4 runs |
surface_max_clades |
40000 |
cap on the clades per surface point |
surface_final_max_clades |
1e5 |
cap on the clades of the direct estimate at the estimate |
surface_seed |
NULL |
seed of the clades; NULL draws one from R's generator, so set.seed() reproduces them |
emphasis_pipeline() conditions on the GAM that its bounds stage trains; leaving that stage out, with control$lower_bound, control$upper_bound and optionally control$survival_gam, conditions on another survival probability or on none.
When the phylogeny samples only a fraction $\rho$ of the living species, set rho in the control list (or simulate_tree(rho = ...)). The augmentation then inserts both extinct lineages and unsampled extant lineages, and the likelihood gains the binomial sampling factor $n_{\text{obs}}\log\rho + n_{\text{unsamp}}\log(1-\rho)$. With rho = 1 (default) the tree is completely sampled.
Both proposals implement this. In the surrogate $p(t)$ becomes the probability of leaving a sampled descendant, which solves the same ODE with the terminal condition $p(t_p) = \rho$ and has the closed form $\rho d / (dE + \rho\lambda(1-E))$ under constant rates; a missing lineage still alive at the present is an unsampled extant tip, emitted with the 5e10 sentinel so that $N(t)$ counts it. The value of rho used is recorded in the fit. A rho outside (0, 1] is an error, in R and in the compiled code alike.
library(emphasis); library(ape)
# a clade under diversity dependence: lambda = exp(b0 + bN N), mu = exp(g0)
sim <- simulate_tree(pars = c(log(0.6), -0.01, log(0.05), 0), max_t = 12,
model = "dd", link = "exponential")
phy <- sim$tes
# 1. one model
box <- auto_bounds(phy, model = "dd", link = "exponential", train_surv_gam = FALSE)
fit <- estimate_rates(phy, model = "dd", link = "exponential",
control = list(lower_bound = box$lower_bound,
upper_bound = box$upper_bound,
sample_size = 200))
fit # estimates, log-likelihood, effective sample, gap, stop reason, notes
# 2. the ladder: cr, dd, nd, ned, nk from nested starts, ranked by AIC
fits <- estimate_rates(phy, model = "all", link = "exponential",
control = list(sample_size = 200))
fits$comparison # AIC, ESS, gap, swing, within_margin; attrs "chosen", "ranking_at_risk"
fits$chosen # the fewest parameters within 2 AIC of the minimum
fits$fits$nd$pars
# 3. the path: which covariates the data ask for first (exponential link only)
cp <- covariate_path(phy, covariates = c("N", "D", "ED", "K"), criterion = "BIC")
cp$speciation$entry_order; cp$speciation$active
# the numbers the likelihood is built on
head(lineage_table(phy))
# the surrogate on a D model (the default there is thinning), thinning anywhere,
# and incomplete sampling
estimate_rates(phy, model = "nd", link = "exponential",
control = list(sampling = "bdi", sample_size = 200,
lower_bound = lb, upper_bound = ub))
estimate_rates(phy, model = "dd", control = list(sampling = "dynamic_fresh", ...))
estimate_rates(phy, model = "dd", control = list(rho = 0.8, ...))
# conditioned on survival through local surfaces of a direct estimate (any model)
fit_c <- estimate_rates(phy, model = "nd", cond = "surface", init_pars = nested_start,
control = list(lower_bound = lb, upper_bound = ub, sample_size = 200))
fit_c$conditioning[c("surfaces", "clades", "seconds", "logP", "logP_se", "start")]
# Forward simulation: lambda(N) = 0.5 exp(-0.02 N), mu = 0.1
sim <- simulate_tree(
pars = c(log(0.5), -0.02, log(0.1), 0),
max_t = 10,
model = "dd",
link = "exponential" # the default; or "linear"
)
sim$status # "done", "extinct", or "too_large"
plot(sim$tes) # reconstructed (extant-only) tree
# Batch simulation (parallel)
pars_mat <- cbind(
beta_0 = log(runif(100, 0.3, 1.0)),
beta_N = runif(100, -0.05, -0.005),
gamma_0 = log(runif(100, 0.01, 0.3)),
gamma_N = rep(0, 100)
)
batch <- simulate_tree(
pars = pars_mat, max_t = 10,
model = "dd", link = "exponential",
max_tries = 1, num_threads = 4
)
# Conditional simulation: augment an observed tree with latent lineages
aug <- simulate_tree(sim$tes, pars = c(log(0.5), -0.02, log(0.1), 0), model = "dd",
n_trees = 50, method = "bdi") # or method = "thinning"
length(aug$trees); aug$log_q
max_tries is the number of forward draws in all, the first included: the default 1 is a single draw, unconditioned on survival, and an extinct clade is redrawn until one survives or max_tries draws have been made.
# fits named one by one, compared with the Monte Carlo guard
fit_cr <- estimate_rates(phy, model = "cr", link = "exponential", control = ctrl_cr)
fit_dd <- estimate_rates(phy, model = "dd", link = "exponential", control = ctrl_dd)
compare_models(cr = fit_cr, dd = fit_dd) # AIC, ESS, gap, swing, within_margin; attr "chosen"
| Function | Purpose |
|---|---|
simulate_tree() |
Forward simulation under every model and link; given a tree, hidden lineages from either proposal (method = "bdi" / "thinning") |
estimate_rates() |
Monte Carlo EM for one model, or the nested ladder with model = "all"; the trace of every iteration in details$mcem |
compare_models() |
AIC table over fits with effective sample, gap, swing and the ranking-at-risk flag; chosen is the fewest parameters within 2 AIC of the minimum, ties broken by the lower AIC |
covariate_path() |
The dgLARS path over the covariates, for speciation and extinction; entry order and the set BIC or AIC chooses |
lineage_table() |
One row per lineage and segment: the covariates and the event mark the likelihood reads |
auto_bounds() |
The parameter box from the observed tree, and the optional survival GAM |
survival_mc() |
$\log P_\theta(S)$ by direct simulation, for any model, link and $\rho$ |
survival_surface() |
A local quadratic surface of $\log P_\theta(S)$; predict() gives $\log P$, $P$ or its gradient |
diagnose_mcem() |
The trace of a fit: iterates, log-likelihood, effective sample, weights |
emphasis_pipeline(), train_GAM(), predict_survival(), diagnose_cem(), diagnose_gam() |
Further estimators and the survival emulator, exported for compatibility |
The full reference, with every argument, is on the docs site.
Each experiment is pre-registered with its rule before it runs, and has one page on the docs site (experiments). The conclusions that hold are collected on What is established and Estimability.
cr it meets the pre-set rule in all 24 cells (25-800 tips; turnover 0.10, 0.25, 0.40; sampling fraction 1 and 0.8). Under dd with complete sampling it meets it in 11 of 14 cells: all nine at 25-100 tips, and two of five at 200-400 tips, where the three failing cells are those whose fits ended on the eight-hour clock (E45).bd_ML also puts extinction at zero on 37 of 40 trees (E32).cr (mean ratio 1.000 over 450 trees) and dd (within 0.02) (V2).182055b the mc_error stopping rule could not fire for d, nd, k and nk fitted with the thinning proposal, so every such fit ran to its iteration cap; fits under the default rel_change rule were not affected.The rest of the earlier results are being remeasured by a validation series, V1-V12, each replacing named earlier experiments: proposal densities (V2), stopping rule and box (V3), exact agreement including 170 real trees (V4), recovery of the eight lineage-level models by tree size under the conditioned likelihood (V5, from E46), sampling fraction (V6), model choice (V7), covariate path (V8), misspecification (V9), extinction (V10), real phylogenies (V11) and cost (V12). Until V5 reports, no lineage-level coefficient has a measured reliable tree size.
| Repository | Holds |
|---|---|
emphasis-paper (private) |
the pre-registrations (00-design.md), drivers (sim/), results (results/) and reports: the validation series and its status (report/validation-series.pdf), the audit of E1-E48 (report/experiment-history.pdf), the map from each E experiment to the V experiment replacing it (report/e-to-v-map.md), the results the papers must state (report/paper-results-2026-10-31.md) |
emphasis-methods, emphasis-biology (private) |
the two manuscripts, the method and the bird and mammal application, each with a supplement |
| experiments wiki (private) | one page per experiment, rendered into the docs site by site/build.py |
The two manuscripts are being revised so that they state only results that hold and point each pending one to its V experiment; both still lack an abstract, and their titles are open.
The test suite has 42 files in tests/testthat/. At the last full run 42,458 expectations passed with NOT_CRAN=true and 2,225 in CRAN mode, where the Monte Carlo gates are skipped; a full run takes a few minutes. CI runs R CMD check on macOS, Windows and Linux (R release, devel and oldrel).
These files hold the likelihood and the proposals to a computation written independently of the C++ code, or to the simulator:
| File | What it checks |
|---|---|
test-d-bookkeeping.R |
the $D$ compensator on a fully observed tree against pendant ages read off the edge list, under both links, and the d/nd complete-data log-likelihood of simulated trees against one computed in R |
test-ed-complete.R |
the raw-$\mathrm{ED}$ complete-data log-likelihood of simulated ed trees against one computed in R from the definition of fair proportion |
test-d-compensator.R |
the linear-link $D$ compensator against quadrature over the instantaneous $D$, and the pendant PD the thinning sampler reads against a replay of the tree |
test-ed.R |
$\mathrm{ED}$ against a fair-proportion computation on ape trees, its sum against Faith's PD, and ned at $\beta_{ED} = 0$ against dd |
test-k.R |
the simulator's $K$ against ape's node depth, and the $K$ likelihood against the Poisson form of the lineage table |
test-lineage-table.R |
the full model's log-likelihood rebuilt from lineage_table() against eval_logf, and the point-process path against dglars |
test-de.R |
the importance-sampling estimate on a fixed tree against a direct calculation, and the effective-sample identities |
test-proposal-density.R |
$E_q[f/q]$ against the brute-force marginal likelihood on a 2-tip tree, and the drawn lifetimes against the density $\log q$ charges (KS test) |
test-parent-support.R |
the parent a hidden birth attaches to, and $E_q[f/q]$ against the brute-force marginal likelihood on 3- and 4-tip trees |
test-bdi-cr.R, test-bdi-d.R |
the surrogate's survival probability against Nee's formula and its zero-variance weights under constant rates; $E_q[f/q]$ under a $D$ model against quadrature |
test-thinning-envelope.R |
that the thinning envelope dominates the proposal rate, against a closed form for the probability of no missing lineage |
test-survival-surface.R |
the direct survival estimate against the exact cr value at $\rho = 1$, $0.7$ and $0.4$; the surface against a direct estimate at its centre and its gradient against its value; end-to-end cond = "surface" fits under cr and d |
test-jensen.R |
the gap against the true bias of a log of a Monte Carlo mean, and the warnings of compare_models() |
Others hold the package's behaviour: test-ultrametric.R (refusal of a non-ultrametric tree), test-time-budget.R (the clock inside the E-step and the M-step), test-compare-chosen.R (the parsimonious choice and its tie-break), test-bdi-damping.R (relaxation of the mean field), test-mstep-start.R (the first simplex sized from the box), test-rho.R (incomplete sampling), test-reproducibility.R (seeding of the C++ samplers) and the rest of tests/testthat/.
dev/audit/ holds an audit of the estimator: 01-map.md documents what the code computes (the weights, fhat, the M-step objective, the MCEM/CEM/GAM stages, the index, time and log conventions) and raises hypotheses, each tested against a closed form, a property or a controlled comparison, with the reproduction scripts in dev/audit/checks/; 02-findings.md carries the confirmed findings ranked by severity and the refuted hypotheses with the reason each is not a defect. The measurements of the estimator on simulated and real trees are on the docs site's experiments pages.
https://franciscorichter.github.io/emphasis/ is generated from its sources. One command rebuilds every page from three inputs: the overview, usage and covariate texts in site/src/*.md and site/src/covariates/*.md, the function reference fragments in site/reference/ (made from man/*.Rd by site/reference.R, run on a machine with R), and a checkout of the experiments wiki, passed on the command line:
python3 site/build.py --wiki <path to the experiments wiki checkout> # pandoc 3 required
It writes the root pages (index.html, usage.html, reference.html, established.html, estimability.html, real-phylogenies.html, open-issues.html, glossary.html), covariates/index.html with one page per covariate (N, D, ED, K), the redirect D.html, experiments/index.html and one experiments/E<n>.html per wiki page, copies the wiki's figures to assets/wiki/, numbers figures and equations, builds each long page's table of contents, and exits non-zero on a broken internal link or a missing image. The built pages are committed, and GitHub Pages serves the root of main (.nojekyll), so a push to main is a deploy. The style is one stylesheet, site/style.css.
Francisco Richter (author and maintainer), [email protected]
Thijs Janzen and Hanno Hildenbrandt contributed to an earlier C++ implementation, parts of which remain in src/ and inst/include/.
MIT. See LICENSE for details.