procgen

Status: Maintenance (expect bug fixes and minor updates)

Procgen Benchmark

[Blog Post] [Paper]

16 simple-to-use procedurally-generated gym environments which provide a direct measure of how quickly a reinforcement learning agent learns generalizable skills. The environments run at high speed (thousands of steps per second) on a single core.

We ran a competition in 2020 which used these environments to measure sample efficiency and generalization in RL. You can learn more here.

These environments are associated with the paper Leveraging Procedural Generation to Benchmark Reinforcement Learning (citation). The code for running some experiments from the paper is in the train-procgen repo. For those familiar with the original CoinRun environment, be sure to read the updated CoinRun description below as there have been subtle changes to the environment.

Compared to Gym Retro, these environments are:

• Faster: Gym Retro environments are already fast, but Procgen environments can run >4x faster.
• Randomized: Gym Retro environments are always the same, so you can memorize a sequence of actions that will get the highest reward. Procgen environments are randomized so this is not possible.
• Customizable: If you install from source, you can perform experiments where you change the environments, or build your own environments. The environment-specific code for each environment is often less than 300 lines. This is almost impossible with Gym Retro.

Supported platforms:

• Windows 10
• macOS 10.14 (Mojave), 10.15 (Catalina)
• Linux (manylinux2010)

Supported Pythons:

• 3.7 64-bit
• 3.8 64-bit
• 3.9 64-bit
• 3.10 64-bit

Supported CPUs:

• Must have at least AVX

Installation

First make sure you have a supported version of python:

…

import gym env = gym.make("procgen:procgen-coinrun-v0", start_level=0, num_levels=1)

Since the gym environment is adapted from a gym3 environment, early calls to `reset()` are disallowed and the `render()` method does not do anything.  To render the environment, pass `render_mode="human"` to the constructor, which will send `render_mode="rgb_array"` to the environment constructor and wrap it in a `gym3.ViewerWrapper`.  If you just want the frames instead of the window, pass `render_mode="rgb_array"`.

For the gym3 vectorized environment:

from procgen import ProcgenGym3Env env = ProcgenGym3Env(num=1, env_name="coinrun", start_level=0, num_levels=1)

To render with the gym3 environment, pass `render_mode="rgb_array"`.  If you wish to view the output, use a `gym3.ViewerWrapper`.

## Saving and loading the environment state

If you are using the gym3 interface, you can save and load the environment state:

from procgen import ProcgenGym3Env env = ProcgenGym3Env(num=1, env_name="coinrun", start_level=0, num_levels=1) states = env.callmethod("get_state") env.callmethod("set_state", states)

…

git clone [email protected]:openai/procgen.git cd procgen conda env update --name procgen --file environment.yml conda activate procgen pip install -e .

this should say "building procgen...done"

python -c "from procgen import ProcgenGym3Env; ProcgenGym3Env(num=1, env_name='coinrun')"

this should create a window where you can play the coinrun environment

python -m procgen.interactive

…

{ struct libenv_tensortype s; strcpy(s.name, "heist_key_count"); s.scalar_type = LIBENV_SCALAR_TYPE_DISCRETE; s.dtype = LIBENV_DTYPE_INT32; s.ndim = 0, s.low.int32 = 0; s.high.int32 = INT32_MAX; info_types.push_back(s); }

This lets the Python code know to expect a single integer and expose it in the `info` dict.

After adding that, you can add the following code to [heist.cpp](https://github.com/openai/procgen/blob/master/procgen/src/games/heist.cpp#L93):

void observe() override { Game::observe(); int32_t key_count = 0; for (const auto& has_key : has_keys) { if (has_key) { key_count++; } } *(int32_t *)(info_bufs[info_name_to_offset.at("heist_key_count")]) = key_count; }

This populates the `heist_key_count` info value each time the environment is observed.

If you run the interactive script (making sure that you installed from source), the new keys should appear in the bottom left hand corner:

`python -m procgen.interactive --env-name heist`

# Changelog

See [CHANGES](CHANGES.md) for changes present in each release.

# Contributing

See [CONTRIBUTING](CONTRIBUTING.md) for information on contributing.

# Assets

See [ASSET_LICENSES](ASSET_LICENSES.md) for asset license information.

# Citation

Please cite using the following bibtex entry:

@article{cobbe2019procgen, title={Leveraging Procedural Generation to Benchmark Reinforcement Learning}, author={Cobbe, Karl and Hesse, Christopher and Hilton, Jacob and Schulman, John}, journal={arXiv preprint arXiv:1912.01588}, year={2019} }