A Swiss-Army Knife for Data I/O

Streamlined data import and export by making assumptions that the user is probably willing to make: 'import()' and 'export()' determine the data structure from the file extension, reasonable defaults are used for data import and export (e.g., 'stringsAsFactors=FALSE'), web-based import is natively supported (including from SSL/HTTPS), compressed files can be read directly without explicit decompression, and fast import packages are used where appropriate. An additional convenience function, 'convert()', provides a simple method for converting between file types.


The aim of rio is to make data file I/O in R as easy as possible by implementing three simple functions in Swiss-army knife style:

  • export() and import() provide a painless data I/O experience by automatically choosing the appropriate data read or write function based on file extension
  • convert() wraps import() and export() to allow the user to easily convert between file formats (thus providing a FOSS replacement for programs like Stat/Transfer or Sledgehammer). Luca Braglia has created a Shiny app called rioweb that provides access to the file conversion features of rio. GREA is an RStudio add-in that provides an interactive interface for reading in data using rio.

rio supports a variety of different file formats for import and export.

FormatImportExport
Tab-separated data (.tsv)YesYes
Comma-separated data (.csv)YesYes
CSVY (CSV + YAML metadata header) (.csvy)YesYes
Feather R/Python interchange format (.feather)YesYes
Pipe-separated data (.psv)YesYes
Fixed-width format data (.fwf)YesYes
Serialized R objects (.rds)YesYes
Saved R objects (.RData)YesYes
JSON (.json)YesYes
YAML (.yml)YesYes
Stata (.dta)YesYes
SPSS (.sav)YesYes
SPSS Portable (.por)Yes
"XBASE" database files (.dbf)YesYes
Excel (.xls)Yes
Excel (.xlsx)YesYes
Weka Attribute-Relation File Format (.arff)YesYes
R syntax (.R)YesYes
Shallow XML documents (.xml)YesYes
HTML Tables (.html)YesYes
SAS (.sas7bdat)YesYes
SAS XPORT (.xpt)Yes
Minitab (.mtp)Yes
Epiinfo (.rec)Yes
Systat (.syd)Yes
Data Interchange Format (.dif)Yes
OpenDocument Spreadsheet (.ods)Yes
Fortran data (no recognized extension)Yes
Google SheetsYes
Clipboard (default is tsv)Yes (Mac and Windows)Yes (Mac and Windows)

Additionally, any format that is not supported by rio but that has a known R implementation will produce an informative error message pointing to a package and import or export function. Unrecognized formats will yield a simple "Unrecognized file format" error.

Because rio is meant to streamline data I/O, the package is extremely easy to use. Here are some examples of reading, writing, and converting data files.

Exporting data is handled with one function, export():

library("rio")
 
export(mtcars, "mtcars.csv") # comma-separated values
export(mtcars, "mtcars.rds") # R serialized
export(mtcars, "mtcars.sav") # SPSS

Importing data is handled with one function, import():

x <- import("mtcars.csv")
y <- import("mtcars.rds")
z <- import("mtcars.sav")
 
# confirm data match
all.equal(x, y, check.attributes = FALSE)
## [1] TRUE
all.equal(x, z, check.attributes = FALSE)
## [1] TRUE

Note: Because of inconsistencies across underlying packages, the data.frame returned by import might vary slightly (in variable classes and attributes) depending on file type.

The convert() function links import() and export() by constructing a dataframe from the imported file and immediately writing it back to disk. convert() invisibly returns the file name of the exported file, so that it can be used to programmatically access the new file.

convert("mtcars.sav", "mtcars.dta")

It is also possible to use rio on the command-line by calling Rscript with the -e (expression) argument. For example, to convert a file from Stata (.dta) to comma-separated values (.csv), simply do the following:

Rscript -e "rio::convert('iris.dta', 'iris.csv')"

The core advantage of rio is that it makes assumptions that the user is probably willing to make. Eight of these are important:

  1. rio uses the file extension of a file name to determine what kind of file it is. This is the same logic used by Windows OS, for example, in determining what application is associated with a given file type. By taking away the need to manually match a file type (which a beginner may not recognize) to a particular import or export function, rio allows almost all common data formats to be read with the same function. Other packages do this as well, but rio aims to be more complete and more consistent than each:
  • reader handles certain text formats and R binary files
  • io offers a set of custom formats
  • ImportExport focuses on select binary formats (Excel, SPSS, and Access files) and provides a Shiny interface.
  • SchemaOnRead iterates through a large number of possible import methods until one works successfully
  1. rio uses data.table::fread() for text-delimited files to automatically determine the file format regardless of the extension. So, a CSV that is actually tab-separated will still be correctly imported. It's also crazy fast.

  2. rio, wherever possible, does not import character strings as factors.

  3. rio stores metadata from rich file formats (SPSS, Stata, etc.) in variable-level attributes in a consistent form regardless of file type or underlying import function. The gather_attrs() function makes it easy to move variable-level attributes to the data frame level.

  4. rio supports web-based imports natively, including from SSL (HTTPS) URLs, from shortened URLs, from URLs that lack proper extensions, and from (public) Google Documents Spreadsheets.

  5. rio imports from from single-file .zip and .tar archives automatically, without the need to explicitly decompress them. Export to compressed directories is also supported.

  6. rio imports and exports files based on an internal S3 class infrastructure. This means that other packages can contain extensions to rio by registering S3 methods. These methods should take the form .import.rio_X() and .export.rio_X(), where X is the file extension of a file type. An example is provided in the rio.db package.

  7. rio wraps a variety of faster, more stream-lined I/O packages than those provided by base R or the foreign package. It uses haven for reading and writing SAS, Stata, and SPSS files, the fread function from data.table for delimited formats, smarter and faster fixed-width file import and export routines, and readxl for reading from Excel workbooks.

The package is available on CRAN and can be installed directly in R using:

install.packages("rio")

The latest development version on GitHub can be installed using devtools:

if(!require("ghit")){
    install.packages("ghit")
}
ghit::install_github("leeper/rio")

News

CHANGES TO v0.4.15

  • Added support for importing from multi-table HTML files using the which argument. (#126)

CHANGES TO v0.4.15

  • Improved behavior of import() and export() with respect to unrecognized file types. (#124, #125, h/t Jason Becker)
  • Added explicit tests of the S3 extension mechanism for .import() and .export().
  • Attempt to recognize compressed but non-archived file formats (e.g., ".csv.gz"). (#123, h/t trevorld)

CHANGES TO v0.4.14

  • Update import and export methods to use new xml2 for XML and HTML export. (#86)

CHANGES TO v0.4.13

  • Fix failing tests related to stricter variable name handling for Stata files in development version of haven. (#113, h/t Hadley Wickham)
  • Added support for export of .sas7bdat files via haven (#116)
  • Restored support for import from SPSS portable via haven (#116)
  • Updated import methods to reflect changed formal argument names in haven. (#116)
  • Converted to roxygen2 documentation and made NEWS an explicit markdown file.

CHANGES TO v0.4.12

  • rio sets options(datatable.fread.dec.experiment=FALSE) during onLoad to address a Unix-specific locale issue.

CHANGES TO v0.4.11

  • Note unsupported NumPy i/o via RcppCNPy. (#112)
  • Fix import of European-style CSV files (sep = "," and sep2 = ";"). (#106, #107, h/t Stani Stadlmann)

CHANGES TO v0.4.10

  • Changed feather Imports to Suggests to make rio installable on older R versions. (#104)
  • Noted new RStudio add-in, GREA, that uses rio. (#109)
  • Migrated CSVY-related code to separate package (https://github.com/leeper/csvy/). (#111)

CHANGES TO v0.4.9

  • Removed unnecessary error in xlsx imports. (#103, h/t Kevin Wright)

CHANGES TO v0.4.8

  • Fixed a bug in the handling of "labelled" class variables imported from haven. (#102, h/t Pierre LaFortune)

CHANGES TO v0.4.7

  • Improved use of the sep argument for import of delimited files. (#99, h/t Danny Parsons)
  • Removed support for import of SPSS Portable (.por) files, given deprecation from haven. (#100)

CHANGES TO v0.4.5

  • Fixed other tests to remove (unimportant) warnings.
  • Fixed a failing test of file compression that was found in v0.4.3 on some platforms.

CHANGES TO v0.4.3

  • Improved, generalized, tested, and expanded documentation of which argument in import().
  • Expanded test suite and made some small fixes.

CHANGES TO v0.4.2

  • Added support to import and export to feather data serialization format. (#88, h/t Jason Becker)

CHANGES TO v0.4.1

  • Fixed behavior of gather_attrs() on a data.frame with no attributes to gather. (#94)
  • Removed unrecognized file format error for import from compressed files. (#93)

CHANGES TO v0.4.0

  • CRAN Release.

CHANGES TO v0.3.19

  • Added a gather_attrs() function that moves variable-level attributes to the data.frame level. (#80)
  • Added preliminary support for import from HTML tables (#86)

CHANGES TO v0.3.18

  • Added support for export to HTML tables. (#86)

CHANGES TO v0.3.17

  • Fixed a bug in import from remote URLs with incorrect file extensions.

CHANGES TO v0.3.16

  • Added support for import from fixed-width format files via readr::read_fwf() with a specified widths argument. This may enable faster import of these types of files and provides a base-like interface for working with readr. (#48)

CHANGES TO v0.3.15

  • Added support for import from and export to yaml. (#83)
  • Fixed a bug when reading from an uncommented CSVY yaml header that contained single-line comments. (#84, h/t Tom Aldenberg)

CHANGES TO v0.3.14

  • Diagnostic messages were cleaned up to facilitate translation. (#57)

CHANGES TO v0.3.12

  • .import() and .export() are now exported S3 generics and documentation has been added to describe how to write rio extensions for new file types. An example of this functionality is shown in the new suggested "rio.db" package.

CHANGES TO v0.3.11

  • import() now uses xml2 to read XML structures and export() uses a custom method for writing to XML, thereby negating dependency on the XML package. (#67)
  • Enhancements were made to import and export of CSVY to store attribute metadata as variable-level attributes (like imports from binary file formats).
  • import() gains a which argument that is used to select which file to return from within a compressed tar or zip archive.
  • Export to tar now tries to correct for bugs in tar() that are being fixed in base R via PR#16716.

CHANGES TO v0.3.10

  • Fixed a bug in import() (introduced in #62, 7a7480e5) that prevented import from clipboard. (h/t Kevin Wright)
  • export() returns a character string. (#82)

CHANGES TO v0.3.9

  • The use of import() for SAS, Stata, and SPSS files has been streamlined. Regardless of whether the haven = TRUE argument is used, the data.frame returned by import() should now be (nearly) identical, with all attributes stored at the variable rather than data.frame level. This is a non-backwards compatible change. (#80)

CHANGES TO v0.3.8

  • Fixed error in export to CSVY with a commented yaml header. (#81, h/t Andrew MacDonald)

CHANGES TO v0.3.7

  • export() now allows automatic file compression as tar, gzip, or zip using the file argument (e.g., export(iris, "iris.csv.zip")).

CHANGES TO v0.3.6

  • Expanded verbosity of export() for fixed-width format files and added a commented header containing column class and width information.
  • Exporting factors to fixed-width format now saves those values as integer rather than numeric.
  • Expanded test suite and separated tests into format-specific files. (#51)

CHANGES TO v0.3.5

  • Export of CSVY files now includes commenting the yaml header by default. Import of CSVY accommodates this automatically. (#74)

CHANGES TO v0.3.3

  • Export of CSVY files and metadata now supported by export(). (#73)
  • Import of CSVY files now stores dataset-level metadata in attributes of the output data.frame. (#73, h/t Tom Aldenberg)
  • When rio receives an unrecognized file format, it now issues a message. The new internal .import.default() and .export.default() then produce an error. This enables add-on packages to support additional formats through new s3 methods of the form .import.rio_EXTENSION() and .export.rio_EXTENSION().

CHANGES TO v0.3.2

  • Use S3 dispatch internally to call new (unexported) .import() and .export() methods. (#42, h/t Jason Becker)

CHANGES TO v0.3.0

  • Release to CRAN.
  • Set a default numerical precision (of 2 decimal places) for export to fixed-width format.

CHANGES TO v0.2.13

  • Import stats package for na.omit().

CHANGES TO v0.2.11

  • Added support for direct import from Google Sheets. (#60, #63, h/t Chung-hong Chan)

CHANGES TO v0.2.7

  • Refactored remote file retrieval into separate (non-exported) function used by import(). (#62)
  • Added test sutie to test file conversion.
  • Expanded test suite to include test of all export formats.

CHANGES TO v0.2.6

  • Cleaned up NAMESPACE file.

CHANGES TO v0.2.5

  • If file format for a remote file cannot be identified from the supplied URL or the final URL reported by curl::curl_fetch_memory(), the HTTP headers are checked for a filename in the Content-Disposition header. (#36)
  • Removed longurl dependency. This is no longer needed because we can identify formats using curl's url argument.
  • Fixed a bug related to importing European-style ("csv2") format files. (#44)
  • Updated CSVY import to embed variable-level metadata. (#52)
  • Use urltools::url_parse() to extract file extensions from complex URLs (e.g., those with query arguments). (#56)
  • Fixed NAMESPACE notes for base packages. (#58)

CHANGES TO v0.2.4

  • Modified behavior so that files imported using haven now store variable metadata at the data.frame level by default (unlike the default behavior in haven, which can cause problems). (#37, h/t Ista Zahn)
  • Added support for importing CSVY (http://csvy.org/) formatted files. (#52)
  • Added import dependency on data.table 1.9.5. (#39)

CHANGES TO v0.2.2

  • Uses the longurl package to expand shortened URLs so that their file type can be easily determined.

CHANGES TO v0.2.1

  • Improved support for importing from compressed directories, especially web-based compressed directories. (#38)
  • Add import dependency on curl >= 0.6 to facilitate content type parsing and format inference from URL redirects. (#36)
  • Add bit64 to Suggests to remove an import warning.

CHANGES TO v0.2

  • import always returns a data.frame, unless setclass is specified. (#22)
  • Added support for import from legacy Excel (.xls) files readxl::read_excel, making its use optional. (#19)
  • Added support for import from and export to the system clipboard on Windows and Mac OS.
  • Added support for export to simple XML documents. (#12)
  • Added support for import from simple XML documents via XML::xmlToDataFrame. (#12)
  • Added support for import from ODS spreadsheet formats. (#12, h/t Chung-hong Chan)
  • Use data.table::fread by default for reading delimited files. (#3)
  • Added support for import and export of dput and dget objects. (#10)
  • Added support for reading from compressed archives (.zip and .tar). (#7)
  • Added support for writing to fixed-width format. (#8)
  • Set stringsAsFactors = FALSE as default for reading tabular data. (#4)
  • Added support for HTTPS imports. (#1, h/t Christopher Gandrud)
  • Added support for automatic file naming in export based on object name and file format. (#5)
  • Exposed convert function.
  • Added vignette, knitr-generated README.md, and updated documentation. (#2)
  • Added some non-exported functions to simplify argument passing and streamline package API. (#6)
  • Separated import, export, convert, and utilities into separate source code files.
  • Expanded the set of supported file types/extensions, switched SPSS, SAS, and Stata formats to haven, making its use optional.

CHANGES TO v0.1.2

  • Updated documentation and fixed a bug in csv import without header.

CHANGES TO v0.1.1

  • Initial release

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("rio")

0.4.16 by Thomas J. Leeper, 5 months ago


https://github.com/leeper/rio


Report a bug at https://github.com/leeper/rio/issues


Browse source code at https://github.com/cran/rio


Authors: Jason Becker [ctb], Chung-hong Chan [aut], Geoffrey CH Chan [ctb], Thomas J. Leeper [aut, cre], Christopher Gandrud [ctb], Andrew MacDonald [ctb], Ista Zahn [ctb], Stanislaus Stadlmann [ctb]


Documentation:   PDF Manual  


Task views: Web Technologies and Services


GPL-2 license


Imports tools, stats, utils, urltools, foreign, haven, openxlsx, readODS, jsonlite, xml2, curl, data.table, readxl, readr, yaml, csvy

Suggests datasets, bit64, testthat, knitr, magrittr, feather


Imported by boxr, dpmr, psData, userfriendlyscience.


See at CRAN