Multi-Agent Reinforcement Learning Chapter 6: Value Iteration for Zero-Sum Games

Live recording of online meeting reviewing material from "Multi-Agent Reinforcement Learning: Foundations and Modern Approaches" by Stefano V. Albrecht, Filippos Christianos, Lukas Schäfer. In this meeting we introduce the first algorithm to find equilibrium solutions in stochastic games. Value iteration uses dynamic programming to iteratively compute an exact solution to stochastic games which have a known equilibrium solution. For zero-sum games, we can use a linear program to solve for a minimax solution in each game state. The algorithm also uses a Bellman style update to compute game reward values for the joint-actions in each state. The combination of the state value function and the game rewards converge to their true values resulting in a solvable non-repeated normal form game for each state.

A simple two-player soccer game is introduced to study minimax algorithms. We first introduce the environment and observe its behavior with random policies. Then value iteration is used to find the unique minimax solution for every state and these policies are used against each other. Finally, we study the performance on the minimax algorithm against a random policy.

I'm using the following repository to store notes and interactive tools for multi-agent reinforcement learning: https://github.com/jekyllstein/MARL_c...

My previous material on reinforcement learning contains complete notes on the Sutton and Barto RL book: https://jekyllstein.github.io/Reinfor...

The textbook website contains materials provided by the authors including a pdf of the text, slides, and a github repository with code.

MARL textbook website: https://www.marl-book.com/
MARL kickoff slides: https://docs.google.com/presentation/...

This online meeting is hosted through https://www.meetup.com/boulderdatasci... and https://www.meetup.com/silicon-valley...

For background material covering traditional reinforcement learning see the following playlist:    • Reinforcement Learning Tutorial Meetings  

Previous meetings have covered the textbook "Reinforcement Learning: An Introduction" by Richard S. Sutton and Andrew G. Barto and the following links relate to that material and my notes/code based on it.

Sutton and Barto Textbook: http://incompleteideas.net/book/the-b...
HTML Notes: https://jekyllstein.github.io/Reinfor...
GitHub Repository: https://github.com/jekyllstein/Reinfo...

Notes and interactive tools seen in those video use the Julia Language (https://julialang.org/) and the package Pluto.jl (https://plutojl.org/).

#reinforcementlearning #education #multiplayergames