Multiple Imputation with Denoising Autoencoders

A tool for multiply imputing missing data using 'MIDAS', a deep learning method based on denoising autoencoder neural networks (see Lall and Robinson, 2022; ). This algorithm offers significant accuracy and efficiency advantages over other multiple imputation strategies, particularly when applied to large datasets with complex features. Alongside interfacing with 'Python' to run the core algorithm, this package contains functions for processing data before and after model training, running imputation model diagnostics, generating multiple completed datasets, and estimating regression models on these datasets. For more information see Lall and Robinson (2023) . This package is deprecated in favor of 'rMIDAS2'; it remains available for existing workflows but will receive only compatibility and documentation updates.

rMIDAS

⚠ Deprecation notice

rMIDAS is deprecated. Please use rMIDAS2, which replaces rMIDAS with a faster PyTorch-based backend, a simpler API (no manual preprocessing), and no reticulate dependency at runtime. rMIDAS will remain on CRAN for existing users but will not receive new features or bug fixes. Source repository: https://github.com/MIDASverse/rMIDAS2

A migration guide is included as a vignette in both packages: vignette("migrating-to-rMIDAS2", package = "rMIDAS").

Install the replacement:
install.packages("rMIDAS2")

Overview

rMIDAS is an R package for accurate and efficient multiple imputation using deep learning methods. The package provides a simplified workflow for imputing and then analyzing data:

convert() carries out all necessary preprocessing steps
train() constructs and trains a MIDAS imputation model
complete() generates multiple completed datasets from the trained model
combine() runs regression analysis across the complete data, following Rubin’s combination rules

rMIDAS is based on the Python package MIDASpy.

Efficient handling of large data

rMIDAS also incorporates several features to streamline and improve the the efficiency of multiple imputation analysis:

Optimisation for large datasets using data.table and mltools packages
Automatic reversing of all pre-processing steps prior to analysis
Built-in regression function based on glm (applying Rubin’s combination rules)

Background and suggested citations

For more information on MIDAS, the method underlying the software, see:

Lall, Ranjit, and Thomas Robinson. 2022. "The MIDAS Touch: Accurate and Scalable Missing-Data Imputation with Deep Learning." Political Analysis 30, no. 2: 179-196. Published version.

Lall, Ranjit, and Thomas Robinson. 2023. "Efficient Multiple Imputation for Diverse Data in Python and R: MIDASpy and rMIDAS." Journal of Statistical Software 107, no. 9: 1-38. doi:10.18637/jss.v107.i09. Published version.

Installation

rMIDAS is available on CRAN. To install the package in R, you can use the following code:

install.packages("rMIDAS")

To install the latest development version, use the following code:

# install.packages("devtools")
devtools::install_github("MIDASverse/rMIDAS")

Note that rMIDAS uses the reticulate package to interface with Python. When the package is first loaded, it will prompt the user on whether to set up a Python environment and its dependencies automatically. Users that choose to set up the environment and dependencies manually, or who use rMIDAS in headless mode can specify a Python binary using set_python_env() (examples below). Currently, Python versions from 3.6 to 3.10 are supported. For a custom Python environment the following dependencies are also required:

matplotlib
numpy
pandas
scikit-learn
scipy
statsmodels
tensorflow (<2.12.0)
tensorflow-addons (<0.20.0)

Setting a custom Python install must be performed before training or imputing data occurs. To manually set up a Python environment:

library(rMIDAS)
# Decline the automatic setup

# Point to a Python binary
set_python_env(x = "path/to/python/binary")

# Or point to a virtualenv binary
set_python_env(x = "virtual_env", type = "virtualenv")

# Or point to a conda environment
set_python_env(x = "conda_env", type = "conda")

# Now run rMIDAS::train() and rMIDAS::complete()...

You can also download the rmidas-env.yml conda environment file from this repository to set up all dependencies in a new conda environment. To do so, download the .yml file, navigate to the download directory in your console and run:

conda env create -f rmidas-env.yml

Then, prior to training a MIDAS model, make sure to load this environment in R:

# First load the rMIDAS package
library(rMIDAS)
# Decline the automatic setup

set_python_env(x = "rmidas", type = "conda")

Note: reticulate only allows you to set a Python binary once per R session, so if you wish to switch to a different Python binary, or have already run train() or convert(), you will need to restart or terminate R prior to using set_python_env().

Vignettes (including simple example)

rMIDAS is packaged with four vignettes:

vignette("imputation_demo", "rMIDAS") demonstrates the basic workflow and capacities of rMIDAS
vignette("custom_python_versions", "rMIDAS") provides detailed guidance on configuring Python binaries and environments, including some troubleshooting tips
vignette("use_server", "rMIDAS") provides guidance for running rMIDAS in headless mode
vignette("migrating-to-rMIDAS2", "rMIDAS") guides migration to the new rMIDAS2 package

An additional example that showcases rMIDAS core functionalities can be found here.

Getting help

rMIDAS is deprecated and is being retained for existing workflows. If you need new development or a simpler installation path, please migrate to rMIDAS2. The successor package source repository is https://github.com/MIDASverse/rMIDAS2. If you encounter an issue that affects an existing rMIDAS workflow, please raise it here.