Calculate Accurate Precision-Recall and ROC (Receiver Operator Characteristics) Curves

Accurate calculations and visualization of precision-recall and ROC (Receiver Operator Characteristics) curves.


The aim of the precrec package is to provide an integrated platform that enables robust performance evaluations of binary classifiers. Specifically, precrec offers accurate calculations of ROC (Receiver Operator Characteristics) and precision-recall curves. All the main calculations of precrec are implemented with C++/Rcpp.

precrec provides accurate precision-recall curves.

  • Non-linear interpolation
  • Elongation to the y-axis to estimate the first point when necessary
  • Use of score-wise threshold values instead of fixed bins

precrec also calculates AUC scores with high accuracy.

precrec calculates curves in a matter of seconds even for a fairly large dataset. It is much faster than most other tools that calculate ROC and precision-recall curves.

In addition to precision-recall and ROC curves, precrec offers basic evaluation measures.

  • Error rate
  • Accuracy
  • Specificity
  • Sensitivity, true positive rate (TPR), recall
  • Precision, positive predictive value (PPV)
  • Matthews correlation coefficient
  • F-score

precrec calculates confidence intervals when multiple test sets are given. It automatically shows confidence bands about the averaged curve in the corresponding plot.

precrec calculates partial AUCs for specified x and y ranges. It can also draw partial ROC and precision-recall curves for the specified ranges.

precrec provides several useful functions that lack in most other evaluation tools.

  • Handling multiple models and multiple test sets
  • Handling tied scores and missing scores
  • Pre- and post-process functions of simple data preparation and curve analysis
  • Install the release version of precrec from CRAN with install.packages("precrec").

  • Alternatively, you can install a development version of precrec from our GitHub repository. To install it:

    1. Make sure you have a working development environment.

      • Windows: Install Rtools (available on the CRAN website).
      • Mac: Install Xcode from the Mac App Store.
      • Linux: Install a compiler and various development libraries (details vary across different flavors of Linux).
    2. Install devtools from CRAN with install.packages("devtools").

    3. Install precrec from the GitHub repository with devtools::install_github("takayasaito/precrec").

The precrec package provides the following five functions.

FunctionDescription
evalmodMain function to calculate evaluation measures
mmdataReformat input data for performance evaluation calculation
join_scoresJoin scores of multiple models into a list
join_labelsJoin observed labels of multiple test datasets into a list
create_sim_samplesCreate random samples for simulations

Moreover, the precrec package provides eight S3 generics for the S3 object created by the evalmod function. N.B. The R language specifies S3 objects and S3 generic functions as part of the most basic object-oriented system in R.

S3 genericPackageDescription
printbasePrint the calculation results and the summary of the test data
as.data.framebaseConvert a precrec object to a data frame
plotgraphicsPlot performance evaluation measures
autoplotggplot2Plot performance evaluation measures with ggplot2
fortifyggplot2Prepare a data frame for ggplot2
aucprecrecMake a data frame with AUC scores
partprecrecCalculate partial curves and partial AUC scores
paucprecrecMake a data frame with pAUC scores
  • Introduction to precrec - a package vignette that contains the descriptions of the functions with several useful examples. View the vignette with vignette("introduction", package = "precrec") in R. The HTML version is also available on the GitPages.

  • Help pages - all the functions including the S3 generics except for print have their own help pages with plenty of examples. View the main help page with help(package = "precrec") in R. The HTML version is also available on the GitPages.

Following two examples show the basic usage of precrec functions.

The evalmod function calculates ROC and Precision-Recall curves and returns an S3 object.

library(precrec)
 
# Load a test dataset
data(P10N10)
 
# Calculate ROC and Precision-Recall curves
sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels)

The autoplot function outputs ROC and Precision-Recall curves by using the ggplot2 package.

# The ggplot2 package is required 
library(ggplot2)
 
# Show ROC and Precision-Recall plots
autoplot(sscurves)

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2016;

doi: 10.1093/bioinformatics/btw570

News

precrec 0.6.1

  • Improve github pages

precrec 0.6

  • Improve as.data.frame with Rcpp

  • Create github pages with pkgdown

precrec 0.5.2

  • Update README

precrec 0.5

  • Add partial AUCs

precrec 0.4

  • Add new measures

    • Matthews correlation coefficient
    • F-score
  • New generic function

    • as.data.frame works on precrec S3 objects

precrec 0.3

  • Improved the testing enviroment

    • unit tests
    • codecov
  • Improved several documents

precrec 0.2

  • Improved several documents
    • several help files (.Rd)
    • package vignette
    • README

precrec 0.1

  • The first release version of precrec

  • The package offers five functions

    • evalmod
    • mmdata
    • join_scores
    • join_labels
    • create_sim_samples

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("precrec")

0.9.1 by Takaya Saito, 3 months ago


http://takayasaito.github.io/precrec, https://github.com/takayasaito/precrec


Report a bug at https://github.com/takayasaito/precrec/issues


Browse source code at https://github.com/cran/precrec


Authors: Takaya Saito [aut, cre], Marc Rehmsmeier [aut]


Documentation:   PDF Manual  


GPL-3 license


Imports Rcpp, ggplot2, assertthat, grid, gridExtra, methods, data.table

Suggests testthat, knitr, rmarkdown

Linking to Rcpp


Imported by HEMDAG, prcbench.


See at CRAN