Statistical Comparison of Multiple Algorithms in Multiple Problems

Given a matrix with results of different algorithms for different problems, the package uses statistical tests and corrections to assess the differences between algorithms.

This is a simple R package aimed at simplifying the statistical analysis of the results in the comparison of algorithms in different problems.

It is mainly focused on non parametric methods and implements Shaffer static and Bergmann and Hommel dynamic corrections for pairwise tests.

The package also includes some plotting tools, such as the critical difference plots shown in Demšar, J., 2006. Indeed, the package is mainly based on the papers:

Demšar, J. (2006) Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7, 1-30.

García S. and Herrera, F. (2008) An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons. Journal of Machine Learning Research, 9, 2677-2694.

García S. and Herrera, F. (2010) Advanced Nonparametric Tests for Multiple Comparison in the Design of Experiments in Computational Intelligence and Data Mining: Experimental Analysis of Power. Information Sciences, 180, 2044-2064.

The last version of the package can be installed running the following commands:

if (!require("devtools")) {

The package can also be installed using the tar.gz files in the root directory. First, download the scmamp_*.tar.gz file and install it running:

install.packages(path.tar.gz.file, reps=NULL)

where path.tar.gz.file refers to the path of the downloaded file. Note that these files may not be up to date.

The package includes two vignettes, one for the basic use and another with information about the data manipulation. To access to the local versions of these vignettes (once the package is installed):



Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.


0.2.55 by Borja Calvo, a year ago

Report a bug at

Browse source code at

Authors: Borja Calvo [aut, cre], Guzman Santafe [aut]

Documentation:   PDF Manual  

GPL-2 license

Imports utils, graphics, ggplot2, graph, reshape2, Rgraphviz, methods

Suggests knitr, rmarkdown

See at CRAN