Visual Regression Testing and Graphical Diffing

An extension to the 'testthat' package that makes it easy to add graphical unit tests. It provides a Shiny application to manage the test cases.


vdiffr is an extension to the package testthat that makes it easy to test for visual regressions. It provides a Shiny app to manage failed tests and visually compare a graphic to its expected output.

Get the development version from github with:

# install.packages("devtools")
devtools::install_github("lionel-/vdiffr")

or the last CRAN release with:

install.packages("vdiffr")

vdiffr requires FreeType greater than 2.6.0. It is automatically installed on Windows along with gdtools and comes with X11 on macOS. If you run an old Linux distribution, it is possible you will have to update the relevant package. See the section on Travis-CI below for some indications.

vdiffr integrates with testthat through the expect_doppelganger() expectation. It takes as arguments:

  • A title. This title is used in two ways. First, the title is standardised (it is converted to lowercase and any character that is not alphanumeric or a space is turned into a dash) and used as filename for storing the figure. Secondly, with ggplot2 figures the title is automatically added to the plot with ggtitle() (only if no ggtitle has been set).

  • A figure. This can be a ggplot object, a recordedplot, a function to be called, or more generally any object with a print method.

  • Optionally, a path where to store the figures, relative to tests/figs/. They are stored in a subfolder according to the current testthat context by default. Supply path to change the subfolder.

For example, the following tests will create figures in tests/figs/histograms/ called base-graphics-histogram.svg and ggplot2-histogram.svg:

context("Histograms")
 
disp_hist_base <- function() hist(mtcars$disp)
disp_hist_ggplot <- ggplot(mtcars, aes(disp)) + geom_histogram()
 
vdiffr::expect_doppelganger("Base graphics histogram", disp_hist_base)
vdiffr::expect_doppelganger("ggplot2 histogram", disp_hist_ggplot)

Note that in addition to automatic ggtitles, ggplot2 figures are assigned the minimalistic theme theme_test() (unless they already have been assigned a theme).

You can run the tests the usual way, for example with devtools::test(). New cases for which you just wrote an expectation will be skipped. Failed tests will show as an error.

When you have added new test cases or detected regressions, you can manage those from the R command line with the functions collect_cases(), validate_cases(), and delete_orphaned_cases(). However it's easier to run the shiny application manage_cases(). With this app you can:

  • Check how a failed case differs from its expected output using three widgets: Toggle (click to swap the images), Slide and Diff. If you use Github, you may be familiar with the last two.

  • Validate cases. You can do so groupwise (all new cases or all failed cases) or on a case by case basis. When you validate a failed case, the old expected output is replaced by the new one.

  • Delete orphaned cases. During a refactoring of your unit tests, some visual expectations may be removed or renamed. This means that some unused figures will linger in the tests/figs/ folder. These figures appear in the Shiny application under the category "Orphaned" and can be cleaned up from there.

Both manage_cases() and collect_cases() take package as first argument, the path to your package sources. This argument has exactly the same semantics as in devtools. You can use vdiffr tools the same way as you would use devtools::check(), for example. The default is ".", meaning that the package is expected to be found in the current folder.

All validated cases are stored in tests/figs/. This folder may be handy to showcase the different graphs offered in your package. You can also keep track of how your plots change as you tweak their layout and add features by checking the history on Github.

An addin to launch manage_cases() is provided with vdiffr. Use the addin menu to launch the Shiny app in an RStudio dialog.

To use the Shiny app as part of ESS devtools integration with C-c C-w C-v, include something like this in your init file:

(defun ess-r-vdiffr-manage-cases ()
  (interactive)
  (ess-r-package-send-process "vdiffr::manage_cases(%s)\n"
                              "Manage vdiffr cases for %s"))
 
(define-key ess-r-package-dev-map "\C-v" 'ess-r-vdiffr-manage-cases)

To work properly, vdiffr requires the C library FreeType version 2.6.0 or later. The FreeType libraries available by default on Travis' Linux platforms are not this recent yet. Some adjustments to the travis.yml file are thus required.

Ubuntu Precise (the default):

addons:
  apt:
    sources:
      - debian-sid
    packages:
      - libfreetype6

Ubuntu Trusty:

sudo: required
before_install: [
  "sudo add-apt-repository \"deb http://archive.ubuntu.com/ubuntu/ xenial main\" -y",
  "sudo apt-get update -q",
  "sudo apt-get install libfreetype6"
]

macOS with XCode 6.1:

osx_image: beta-xcode6.1
disable_homebrew: true
latex: false

macOS with XCode 7.2:

osx_image: xcode7.2
brew_packages: cairo
latex: false

Appveyor does not require any configuration since FreeType 2.6.0 is automatically installed on this platform along with gdtools. However, Fontconfig builds a cache of all system fonts the first time it is run, which can take a while. It is a good idea to add the following in a fontconfig-helper.R testthat file in order to speed up the cache building on Appveyor and on CRAN's Windows servers:

on_appveyor <- function() {
  identical(Sys.getenv("APPVEYOR"), "True")
}
on_cran <- function() {
  !identical(Sys.getenv("NOT_CRAN"), "true")
}
 
# Use minimal fonts.conf to speed up fc-cache
if (on_appveyor() || on_cran()) {
  gdtools::set_dummy_conf()
}

vdiffr currently uses svglite to save the plots in a text format that makes it easy to perform comparisons. This makes the test cases dependent on that package as the SVG translation of the plot may change across different versions of svglite (though that should not happen often). For this reason, whenever you validate a graphical test case, the tests/figs/deps.txt file is updated with a note containing the svglite version. This works the same way as the roxygen version note.

Your graphics might be dependent on other packages besides svglite. If your package is an extension to ggplot2 for instance, the appearance of your plot may change as ggplot2 evolves (as with the 2.0 version which tweaked the grayness of the background color among other changes). For this reason, expect_doppelganger() adds a dependence on ggplot2 when you supply a ggplot2 object. You can also manually add a dependency on any other package by calling vdiffr::add_dependency() anywhere in a test file.

vdiffr extends testthat through a custom Reporter. Reporters are classes (R6 classes in recent versions of testthat) whose instances collect cases and output a summary of the tests. While reporters are usually meant to provide output for the end user, you can also use them in functions to interact with testthat.

vdiffr has a special reporter that does nothing but activate a collecter for the visual test cases. collect_cases() calls devtools::test() with this reporter. When expect_doppelganger() is called, it first checks whether the case is new or failed. If that's the case, and if it finds that vdiffr's collecter is active, it calls the collecter, which in turns records the current test case.

This enables the user to run the tests with the usual development tools and get feedback in the form of skipped or failed cases. On the other hand, when vdiffr's tools are called, we collect information about the tests of interest and wrap them in a data structure.

Comparing SVG files is convenient and should work correctly in most situations. However, SVG is not suitable for tracking really subtle changes and regressions. See vdiffr's issue #1 for a discussion on this. vdiffr may gain additional comparison backends in the future to make the tests more stringent.

News

vdiffr 0.1.1

  • expect_doppelganger() no longer throws an error when FreeType is too old. Instead, the test is skipped. This ensures that R CMD check passes on those platforms (e.g., CRAN's Solaris test server).

  • Depends on gdtools 0.1.2 or later as this version fixes a crash on Linux platforms.

  • widget_toggle(), widget_slide() and widget_diff() now take plots as arguments. This makes it easy to embed a vdiffr widget in R Markdown documents. The underscored versions take HTML sources as argument (paths to SVG files or inline SVGs).

vdiffr 0.1.0

  • Generated SVGs are now reproducible across platforms thanks to recent versions of svglite, gdtools, and the new package fontquiver. vdiffr now requires versions of FreeType greater than 2.6.1.

  • The figures folder is hardcoded to tests/figs/.

  • The figures are now stored in subfolders according to the current testthat context. expect_doppelganger() accepts the path argument to bypass this behaviour (set it to "" to store the figures in tests/figs/).

  • The title argument of expect_doppelganger() now serves as ggtitle() in ggplot2 figures (unless a title is already set). It is also standardised and used as filename to store the figure (spaces and non-alphanumeric characters are converted to dashes).

  • Add support for handling orphaned cases: you can now remove figures left over from deleted tests with delete_orphaned_cases() or from the Shiny app.

  • New filter argument to collect_cases() and manage_cases(). This lets you filter the test files from which to collect the cases, which is useful to speed up the collection for large codebases with a lot of unit tests.

  • Fix invalid generation of SVG files (#3)

  • Give a warning when multiple doppelgangers have the same name (#4).

  • Remove CR line endings before comparing svg files for compatibility with Windows

vdiffr 0.0.0.9000

Initial release

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("vdiffr")

0.2.0 by Lionel Henry, 2 months ago


Browse source code at https://github.com/cran/vdiffr


Authors: Lionel Henry [cre, aut], RStudio [cph], Carl Sutherland [aut] (jg-imagediff library), Humble Software [cph] (jg-imagediff library), David Hong [aut] (TwoFace library), jQuery Foundation [cph] (jQuery library), jQuery contributors [ctb, cph] (jQuery library; authors listed in inst/htmlwidgets/lib/jquery-authors.txt)


Documentation:   PDF Manual  


GPL-3 | file LICENSE license


Imports devtools, fontquiver, gdtools, glue, grDevices, htmlwidgets, purrr, rlang, R6, Rcpp, shiny, svglite, testthat, xml2

Suggests ggplot2, roxygen2, rstudioapi, yaml

Linking to Rcpp

System requirements: FreeType >= 2.6.0


Suggested by atlantistools, ggExtra, ggjoy, ggridges, ggstance, naniar, viridis, visdat.


See at CRAN