AI-Toolbox

AI-Toolbox

This C++ toolbox is aimed at representing and solving common AI problems, implementing an easy-to-use interface which should be hopefully extensible to many problems, while keeping code readable.

Current development includes MDPs, POMDPs and related algorithms. This toolbox was originally developed taking inspiration from the Matlab MDPToolbox, which you can find here, and from the pomdp-solve software written by A. R. Cassandra, which you can find here.

If you are new to the field of reinforcement learning, we have a few simple tutorials that can help you get started. An excellent, more in depth introduction to the basics of reinforcement learning can be found freely online in this book.

If you use this toolbox for research, please consider citing our JMLR article:

@article{JMLR:v21:18-402,
  author  = {Eugenio Bargiacchi and Diederik M. Roijers and Ann Now\'{e}},
  title   = {AI-Toolbox: A C++ library for Reinforcement Learning and Planning (with Python Bindings)},
  journal = {Journal of Machine Learning Research},
  year    = {2020},
  volume  = {21},
  number  = {102},
  pages   = {1-12},
  url     = {http://jmlr.org/papers/v21/18-402.html}
}

Example

…

Documentation

The latest documentation is available here. We have a few tutorials that can help you get started with the toolbox. The tutorials are in C++, but the examples folder contains equivalent Python code which you can follow along just as well.

For Python docs you can find them by typing help(AIToolbox) from the interpreter. It should show the exported API for each class, along with any differences in input/output.

Features

Cassandra POMDP Format Parsing

Cassandra's POMDP format is a type of text file that contains a definition of an MDP or POMDP model. You can find some examples here. While it is absolutely not necessary to use this format, and you can define models via code, we do parse a reasonable subset of Cassandra's POMDP format, which allows to reuse already defined problems with this library. Here's the docs on that.

Python 2 and 3 Bindings!

The user interface of the library is pretty much the same with Python than what you would get by using simply C++. See the examples folder to see just how much Python and C++ code resemble each other. Since Python does not allow templates, the classes are binded with as many instantiations as possible.

Additionally, the library allows the usage of native Python generative models (where you don't need to specify the transition and reward functions, you only sample next state and reward). This allows for example to directly use OpenAI gym environments with minimal code writing.

That said, if you need to customize a specific implementation to make it perform better on your specific use-cases, or if you want to try something completely new, you will have to use C++.

Utilities

The library has an extensive set of utilities which would be too long to enumerate here. In particular, we have utilities for combinatorics, polytopes, linear programming, sampling and distributions, automated statistics, belief updating, many data structures, logging, seeding and much more.

Bandit/Normal Games:

	Models
Basic Model
	Policies
Exploring Selfish Reinforcement Learning (ESRL)	Q-Greedy Policy	Softmax Policy
Linear Reward Penalty	Thompson Sampling (Student-t distribution)	Random Policy
Top-Two Thompson Sampling (Student-t distribution)	Successive Rejects	T3C (Normal distribution)

Single Agent MDP/Stochastic Games:

	Models
Basic Model	Sparse Model	Maximum Likelihood Model
Sparse Maximum Likelihood Model	Thompson Model (Dirichlet + Student-t distributions)
	Algorithms
Dyna-Q	Dyna2	Expected SARSA
Hysteretic Q-Learning	Importance Sampling	Linear Programming
Monte Carlo Tree Search (MCTS)	Policy Evaluation	Policy Iteration
Prioritized Sweeping	Q-Learning	Double Q-Learning
[Q(λ)][qlam]	[R-Learning][rlea]	[SARSA(λ)][sarl]
[SARSA][sars]	[Retrace(λ)][retl]	[Tree Backup(λ)][trel]
[Value Iteration][vait]
	Policies
[Basic Policy][mpol]	[Epsilon-Greedy Policy][megr]	[Softmax Policy][msof]
[Q-Greedy Policy][mqgr]	[PGA-APP][pgaa]	[Win or Learn Fast Policy Iteration (WoLF)][wolf]