Model-Free Reinforcement Learning

Performs model-free reinforcement learning in R. This implementation enables the learning of an optimal policy based on sample sequences consisting of states, actions and rewards. In addition, it supplies multiple predefined reinforcement learning algorithms, such as experience replay.


Reinforcement Learning

Build Status CRAN_Status_Badge

ReinforcementLearning performs model-free reinforcement learning in R. This implementation enables the learning of an optimal policy based on sample sequences consisting of states, actions and rewards. In addition, it supplies multiple predefined reinforcement learning algorithms, such as experience replay.

Overview

The most important functions of ReinforcementLearning are:

  • Learning an optimal policy from a fixed set of a priori known transition samples
  • Predefined learning rules and action selection modes
  • A highly customizable framework for model-free reinforcement learning tasks

Installation

You can easily install the latest version of ReinforcementLearning with

install.packages("ReinforcementLearning")
 
# Alternatively, install the development version from GitHub:
# install.packages("devtools")
devtools::install_github("nproellochs/ReinforcementLearning")

Usage

This section shows the basic functionality of how to perform reinforcement learning. First, load the corresponding package ReinforcementLearning.

library(ReinforcementLearning)

The following example shows how to learn a reinforcement learning agent using input data in the form of sample sequences consisting of states, actions and rewards. The result of the learning process is a state-action table and an optimal policy that defines the best possible action in each state.

# Generate sample experience in the form of state transition tuples
data <- sampleGridSequence(N = 1000)
head(data)
#>   State Action Reward NextState
#> 1    s4   left     -1        s4
#> 2    s2  right     -1        s3
#> 3    s2  right     -1        s3
#> 4    s3   left     -1        s2
#> 5    s4     up     -1        s4
#> 6    s1   down     -1        s2
 
# Define reinforcement learning parameters
control <- list(alpha = 0.1, gamma = 0.1, epsilon = 0.1)
 
# Perform reinforcement learning
model <- ReinforcementLearning(data, s = "State", a = "Action", r = "Reward", 
                               s_new = "NextState", control = control)
 
# Print result
print(model)
#> State-Action function Q
#>          right        up        down       left
#> s1 -1.09619438 -1.098533 -1.00183072 -1.0978962
#> s2 -0.01980279 -1.097758 -1.00252228 -1.0037977
#> s3 -0.02335524  9.884394 -0.01722548 -0.9985081
#> s4 -1.09616040 -1.106392 -1.10548631 -1.1059655
#> 
#> Policy
#>      s1      s2      s3      s4 
#>  "down" "right"    "up" "right" 
#> 
#> Reward (last iteration)
#> [1] -263

License

ReinforcementLearning is released under the MIT License

Copyright (c) 2017 Nicolas Pröllochs & Stefan Feuerriegel

News

ReinforcementLearning 1.0.1

Version 1.0.1

Changes

  • Fixed Tic-Tac-Toe example

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("ReinforcementLearning")

1.0.2 by Nicolas Proellochs, 2 months ago


Browse source code at https://github.com/cran/ReinforcementLearning


Authors: Nicolas Proellochs [aut, cre], Stefan Feuerriegel [aut]


Documentation:   PDF Manual  


MIT + file LICENSE license


Imports ggplot2, hash, data.table

Suggests testthat, knitr, rmarkdown


See at CRAN