Fast Bayesian Methods for AB Testing

A suite of functions that allow the user to analyze A/B test data in a Bayesian framework. Intended to be a drop-in replacement for common frequentist hypothesis test such as the t-test and chi-sq test.


Travis-CI Build Status codecov CRAN_Status_Badge

Fast Bayesian Methods for AB Testing

bayesAB provides a suite of functions that allow the user to analyze A/B test data in a Bayesian framework. bayesAB is intended to be a drop-in replacement for common frequentist hypothesis test such as the t-test and chi-sq test.

Bayesian methods provide several benefits over frequentist methods in the context of A/B tests - namely in interpretability. Instead of p-values you get direct probabilities on whether A is better than B (and by how much). Instead of point estimates your posterior distributions are parametrized random variables which can be summarized any number of ways.

While Bayesian AB tests are still not immune to peeking in the broadest sense, you can use the 'Posterior Expected Loss' provided in the package to draw conclusions at any point with respect to your threshold for error.

The general bayesAB workflow is as follows:

  • Decide how you want to parametrize your data (Poisson for counts of email submissions, Bernoulli for CTR on an ad, etc.)
  • Use our helper functions to decide on priors for your data (?bayesTest, ?plotDistributions)
  • Fit a bayesTest object
    • Optional: Use combine to munge together several bayesTest objects together for an arbitrary / non-analytical target distribution
  • print, plot, and summary to interpret your results
    • Determine whether to stop your test early given the Posterior Expected Loss in summary output

Optionally, use banditize and/or deployBandit to turn a pre-calculated (or empty) bayesTest into a multi-armed bandit that can serve recipe recommendations and adapt as new data comes in.

Note, while bayesAB was designed to exploit data related to A/B/etc tests, you can use the package to conduct Bayesian analysis on virtually any vector of data, as long as it can be parametrized by the available functions.

Installation

Get the latest stable release from CRAN:

install.packages("bayesAB")

Or the dev version straight from Github:

install.packages("devtools")
devtools::install_github("frankportman/bayesAB", build_vignettes = TRUE)

Usage

Some useful links from my blog with bayesAB examples (and pictures!!):

For a more in-depth look please check the package vignettes with browseVignettes(package = "bayesAB") or the pre-knit HTML version on CRAN here. Brief example below. Run the following code for a quick overview of bayesAB:

library(bayesAB)
 
# Choose bernoulli test priors
plotBeta(2, 3)
 

image

# Choose normal test priors
plotInvGamma(12, 4)
 

image

A_binom <- rbinom(100, 1, .5)
B_binom <- rbinom(100, 1, .55)
 
# Fit bernoulli and normal tests
AB1 <- bayesTest(A_binom,
                 B_binom,
                 priors = c('alpha' = 1, 'beta' = 1),
                 distribution = 'bernoulli')
 
plot(AB1)
 

image image image

print(AB1)
 
--------------------------------------------
Distribution used: bernoulli 
--------------------------------------------
Using data with the following properties: 
           A    B
Min.    0.00 0.00
1st Qu. 0.00 0.00
Median  1.00 0.00
Mean    0.55 0.44
3rd Qu. 1.00 1.00
Max.    1.00 1.00
--------------------------------------------
Priors used for the calculation: 
$alpha
[1] 1

$beta
[1] 1

--------------------------------------------
Calculated posteriors for the following parameters: 
Probability 
--------------------------------------------
Monte Carlo samples generated per posterior: 
[1] 1e+05

summary(AB1)
 
Quantiles of posteriors for A and B:

$Probability
$Probability$A
       0%       25%       50%       75%      100% 
0.3330638 0.5159872 0.5496165 0.5824940 0.7507997 

$Probability$B
       0%       25%       50%       75%      100% 
0.2138149 0.4079403 0.4407221 0.4742673 0.6369742 


--------------------------------------------

P(A > B) by (0)%: 

$Probability
[1] 0.93912

--------------------------------------------

Credible Interval on (A - B) / B for interval length(s) (0.9) : 

$Probability
         5%         95% 
-0.01379425  0.58463290 

--------------------------------------------

Posterior Expected Loss for choosing B over A:

$Probability
[1] 0.03105786

News

bayesAB 1.1.0

Breaking

  • Changed conjugate prior of Normal/LogNormal distributions to be the NormalInverseGamma distribution from a combination of the Normal and Inverse Gamma distributions. This distribution is bivariate and gives us a 2d estimate for both x and sig_sq. The params for this distribution are mu, lambda, alpha, beta and are different from the old priors that Normal/LogNormal were expecting.

    • Various doc changes to illustrate these changes and new expectations

Major Changes

  • Fix closed form distributions and added tests
  • Calculation Posterior Expected Loss is now correct and represents a true loss function
  • Added plotNormalInvGamma

Minor Changes

  • Colors for sample plots are now hardcoded (red for > 0 and blue for < 0)
  • Plots are truncated at the extremes to avoid very long tails

bayesAB 1.0.0

Major Additions

  • Added grab and rename to retrieve and rename posteriors from your bayesTest object

    • Mostly useful in conjunction with combine in order to quickly chain together several bayesTests
  • Correctly hide legend for generic plots

  • Standardized prior parameters to have the same arguments as the plot{Dist} functions

    • This mostly changes prior inputs for bayesTest(distribution = c('normal', 'lognormal'))
    • This is a breaking change

Major Internal Changes

  • Moved distribution metadata from bayesTest$distribution to bayesTest$inputs$distribution to be consistent
  • Reconcile posterior names to always be A and B and not include the parameter name
  • A_data and B_data in inputs are now always lists by default to make combine work more simply
  • Big refactor of how bayesTest works internally. Dispatch per distribution is now only related to how the posterior is calculated.
  • Some error checking has been made more generic

Minor Tweaks/Fixed

  • Posterior Expected Loss now correctly displays 0 instead of NaN for that case
  • Numerous doc/examples/tests cleanup
  • Overall refactor of some methods, making it easier to read and contribute

bayesAB 0.7.0

Major Additions

  • added banditize and deployBandit to turn your bayesTest object into a Bayesian multi*armed bandit and deploy as a JSON API respectively.

  • Added programmatic capabilities on top of existing interactive uses for plot generic function

    • You can now assign plot(bayesTestObj) to a variable and not have it automatically plot.
  • Added quantile summary of calculated posteriors to the output of summary.bayesTest

  • Added Posterior Expected Loss to output of summary.bayesTest

    • This is useful to know when to stop your Bayesian AB Test
    • Supports the risk of choosing 'B' over 'A' (ordering is important) and makes more sense if A > B currently in the test

Minor Tweaks/Fixes

  • outputs from plot generics are now explicitly ggplot objects and can be modified as such

    • You can input your own titles/axis labels/etc if the defaults don't fit your use case

bayesAB 0.6.0

Major Additions

  • First major CRAN release
  • 6 (+ 2) distributions
  • print, plot, summary generics
  • Easy plotting of distributions for quick visual inspection
  • combine tests as needed
  • 100% code coverage

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("bayesAB")

1.1.0 by Frank Portman, 9 months ago


https://github.com/FrankPortman/bayesAB


Report a bug at https://github.com/FrankPortman/bayesAB/issues


Browse source code at https://github.com/cran/bayesAB


Authors: Frank Portman [aut, cre]


Documentation:   PDF Manual  


MIT + file LICENSE license


Imports Rcpp, ggplot2, methods

Suggests testthat, knitr, rmarkdown, magrittr, plumber

Linking to Rcpp


See at CRAN