An integrated workflow for factorial survey (vignette) experiments.
Constructs D-efficient sets of vignettes from a factorial structure and
reports their D- and A-efficiency following Kuhfeld, Tobias and Garratt
(1994)
An integrated workflow for factorial survey (vignette) experiments in R.
A factorial survey shows respondents short descriptions whose attributes are
experimentally varied, and asks for a rating -- typically on a bounded scale
such as 0 to 10 -- or a yes/no answer. Because the attributes are randomised by design, their
effects on the rating carry a causal interpretation. FactorialSurvey covers
the stages of such a study in one place: building an efficient set of
vignettes, checking how many respondents are needed, estimating a model suited
to bounded ratings or yes/no answers with standard errors clustered by
respondent, and translating the estimates into quantities that can be
reported: marginal willingness to pay for each attribute and expected
willingness to pay for complete policy scenarios.
See vignette("FactorialSurvey") for a walk-through with real data.
# install.packages("remotes")
remotes::install_github("tanakak/FactorialSurvey")
fs_defficient() picks a D-efficient subset of the full factorial and splits
it into blocks, one per respondent group.
library(FactorialSurvey)
attributes <- list(
damage = c("minor", "moderate", "severe"),
distance = c("near", "far"),
cost = c(100, 500, 1000, 3000)
)
design <- fs_defficient(
attributes,
n_vignettes = 24,
n_blocks = 4,
seed = 2026,
verbose = FALSE
)
design
Combinations that would confuse a respondent can be ruled out before the search:
design <- fs_defficient(
attributes,
n_vignettes = 24,
n_blocks = 4,
restrictions = ~ damage == "minor" & cost == 3000,
seed = 2026,
verbose = FALSE
)
fs_design_eval() reports the D- and A-efficiency (100 percent for an
orthogonal array), how evenly the levels appear and how strongly the
attributes are correlated. fs_efficiency() gives the efficiency of any
design, including one built elsewhere.
fs_design_eval(design)
Before fielding, fs_power() simulates the survey under effects you expect
and reports how often each would be detected, and how precise willingness to
pay would be, for several sample sizes.
set_labels() attaches the wording the respondent will actually see:
design <- set_labels(
as.data.frame(design),
list(damage = c("Minor", "Moderate", "Severe"),
distance = c("Nearby", "Far away"))
)
fs_vignette_text() fills a template with each row's levels. Formatting
functions render numbers the way respondents should read them, without
altering the design columns.
design <- fs_vignette_text(
design,
"A {{damage}} damaged building {{distance}}. Repairs cost {{cost}} a year.",
format = list(cost = function(x) paste0("$", formatC(x, big.mark = ",")))
)
fs_assign() deals blocks out to respondents and orders the vignettes within
each block using a Williams design, so that no attribute is confounded with
position or with the vignette a respondent happened to see just before.
schedule <- fs_assign(design, n_respondents = 200, seed = 1)
fs_export() writes the result out. CSV files carry a UTF-8 byte order mark
by default, which is what makes non-ASCII text open correctly in Excel on
Windows.
fs_export(design, "vignettes.csv")
fs_export(design, "loop.csv", format = "qualtrics") # one file per block
fs_export(schedule, "schedule.xlsx", format = "xlsx") # one sheet per block
fs_betareg() maps the rating scale onto the open unit interval and fits a
beta regression, which respects the bounds of the scale instead of assuming
an unbounded normal error.
fit <- fs_betareg(rating ~ damage + distance + cost,
data = survey_data,
scale = c(0, 10),
cluster = ~ respondent_id)
summary(fit)
Because each respondent rates several vignettes, the ratings of one person
are not independent. cluster makes standard errors, marginal effects and
willingness-to-pay intervals robust to this; without it fs_betareg() warns.
Average marginal effects are reported both on the unit interval and rescaled back onto the original 0--10 scale, so they can be read as "this attribute moves the rating by so many points".
w <- fs_mwtp(fit, price = "cost")
plot(w)
fs_mwtp(fit, price = "cost", method = "krinsky-robb", seed = 1)
# Expected willingness to pay for complete scenarios
scenarios <- data.frame(damage = c("minor", "severe"),
distance = c("far", "near"),
row.names = c("mild", "serious"))
fs_ewtp(fit, scenarios, price = "cost")
MWTP is a ratio of coefficients, so its sampling distribution is skewed. Three interval methods are offered -- the delta method, Krinsky-Robb simulation and Fieller's theorem -- and it is worth comparing them before reporting a result.
| Stage | Functions |
|---|---|
| Design | fs_defficient(), fs_efficiency(), fs_design_eval() |
| Planning | fs_simulate_ratings(), fs_power() |
| Fielding | set_labels(), fs_vignette_text(), fs_williams(), fs_assign(), fs_export() |
| Estimation | fs_betareg() (ratings), fs_logit() (yes/no) |
| Willingness to pay | fs_mwtp(), fs_ewtp() |
| Graphics | plot() methods, theme_fs() |
| Data | bento |
Planned for future releases:
FactorialSurvey is about rating-based vignette experiments. For
choice-based designs and analyses see
idefix and
cregg. The efficient design
search here is built on
AlgDesign and the estimation
on betareg.
MIT