Loose taxonomy of reinforcement learning algorithms
Loose taxonomy of reinforcement learning algorithms
This is a loose taxonomy of reinforcement learning algorithms. I'm by no means expert in this area, I'm making this as part of my learning process. Note that there are a lot more algorithms than listed here, and often I don't even know how to categorize them. In any case, please PR to correct things or suggest new stuff.
Note that this file is generated by taxonomy.py.
Below is the taxonomy of reinforcement learning algorithms. Solid line indicates some progression from one idea to another. Dashed line indicates a loose connection. On the bottom you can see the timeline of the publication year of the algorithms.
It's recommended to open the .SVG file in a new window, as hovering the mouse over the algorithm will show tooltip containing the description of the algorithm and clicking the node will open the link to its description.
Reinforcement learning (RL) is an area of machine learning concerned with how software agents ought to take actions in an environment in order to maximize the notion of cumulative reward [from Wikipedia]
(Path: Reinforcement Learning --> Model Free)
In model free reinforcement learning, the agent directly tries to predict the value/policy without having or trying to model the environment
(Path: Reinforcement Learning --> Model Free --> Value Gradient)
The algorithm is learning the value function of each state or state-action. The policy is implicit, usually by just selecting the best value
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> SARSA)
SARSA (State-Action-Reward-State-Action) is an on-policy TD control method
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> Q-learning)
Q-learning an off-policy TD control method. Unlike SARSA, it doesn't follow the policy to find the next action but rather chooses most optimal action in a greedy fashion
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> TD-Gammon)
TD-Gammon is a model-free reinforcement learning algorithm similar to Q-learning, and uses a multi-layer perceptron with one hidden layer as the value function approximator. It learns the game entirely by playing against itself and achieves superhuman level of play.
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> DQN)
Deep Q Network (DQN) is Q-Learning with deep neural network as state-action value estimator and uses a replay buffer to sample experiences from previous trajectories to make learning more stable.
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> DRQN)
Deep Recurrent Q-Learning. Adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> DDQN)
Double DQN adds another neural network, making separate network for policy and target. The target network is only updated after certain number of steps/episodes. This makes the learning more stable.
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> PER)
Prioritized Experience Replay (PER) improves data efficiency by replaying transitions from which there is more to learn more often
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> Duelling-DQN)
Duelling DQN represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm.
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> QR-DQN)
Distributional Reinforcement Learning with Quantile Regression (QR-DQN). In QR-DQN, distribution of values values are used for each state-action pair instead of a single mean value
(Path: Reinforcement Learning --> Model Free --> Value Gradient --> C51)
C51 Algorithm. The core idea of Distributional Bellman is to ask the following questions. If we can model the Distribution of the total future rewards, why restrict ourselves to the expected value (i.e. Q
No open issues yet, or sync has not completed.