在使用 ICM Wrapper 时,学习率不会衰减在 examples/vizdoom_ppo.py 中

In order to test whether the learning rate decay is actually working, I added a train_fn to the example which tracks the ppo policy learning rate. The git diff is here (can be applied with git apply): tianshou_vizdoom_ppo_no_extend.txt I ran (call this run_grey) the code with the following command (only to make the training resource/time consumption smaller and ensure ICM gets used): Python vizdoom_ppo.py --buffer_size 10000 --epoch 5 --icm_lr_scale 10 --task "D2_navigation"

The primary change (adding the train_fn to track the learning rates in the nested ppo policy and the top level icm policy) is copied below:

    def train_fn(curr_epoch, env_step):
        if ppo.lr_schedulers:
            lr = ppo.lr_schedulers[0].get_last_lr()[0]
            logger.writer.add_scalar("training/lr", lr, global_step=env_step)
        else:
            logger.writer.add_scalar("training/no_ppo_lr", 0, global_step=env_step)
        if algorithm.lr_schedulers:
            for idx, scheduler in enumerate(algorithm.lr_schedulers):
                lr = scheduler.get_last_lr()[0]
                logger.writer.add_scalar(f"training/algorithm_lr_{idx}", lr, global_step=env_step)
        else:
            logger.writer.add_scalar("training/no_algorithm_lr", 0, global_step=env_step)

My hypothesis is that the scheduler attached to the ppo policy never gets stepped because:

1. the top level ICM policy itself does not have any lr_schedulers - it does not check for lr_schedulers in wrapped algorithms, and it's update method calls update_with_batch for the wrapped algorithm (in OnPolicyWrapperAlgorithm in algorithm_base.py) rather than update
2. the lr_schedulers are only stepped in the _update method (in Algorithm class in algorithm_base.py)

And based on my hypothesis I added the wrapped_algorithm's scheduler to the top level (ICM) policy:

algorithm.lr_schedulers.extend(algorithm.wrapped_algorithm.lr_schedulers)

The git patch with this change is here: tianshou_vizdoom_ppo_with_extend.txt And ran the code again with the same command (call this run_pink).

And now in tensorboard I can see the learning rate decreasing. Note: In the attached images grey colour is for the run_grey run, and the pink colour is for the run_pink run. According to the train_fn code: 1. if the ppo algorithm itself does not have a learning rate scheduler attached, a training/no_ppo_lr graph should be plotted - no such graph is plotted in either run - meaning in both runs there is a scheduler attached to the ppo algorithm - and correspondingly the training/lr plot shows up in both runs: As we can see, in pink, the learning rate