[Bug]: Rendering with EvalCallback does not render the initial or final state
Bug
EvalCallbackdoes not render initial state of the episode. This is an issue for environments which have dynamic/random initial states, as it makes it unclear to the user what the state looked like before the agent performed its first step.EvalCallbackdoes not render the last step of the episode, instead rendering the reset state. This robs the user of seeing the last (often crucial) step and is confusing for new users.
This behavior is because the environment reset is performed within the DummyVecEnv.step function. I have a local implementation that fixes this issue and clearly renders both initial and final states, but I'd first like to confirm that this behavior should indeed be treated as a bug worthy of a feature/fix.
To Reproduce
from typing import Optional
import numpy as np
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3 import DQN
from stable_baselines3.common.monitor import Monitor
from stable_baselines3.common.callbacks import EvalCallback
from stable_baselines3.common.envs import BitFlippingEnv
# A user should not be required to subclass their environment to debug the EvalCallback
class ClearerRenderEnv(BitFlippingEnv):
def step(self, *args, **kwargs):
pre_state = self.state.copy()
rets = super().step(*args, **kwargs)
print(f"step : {pre_state} -> {self.state} | {self.desired_goal}")
return rets
def reset(self, *args, **kwargs):
if not hasattr(self, "state"):
return super().reset(*args, **kwargs)
pre_state = self.state.copy()
rets = super().reset(*args, **kwargs)
print(f"reset : {pre_state} -> {self.state} | {self.desired_goal} ")
return rets
def render(self) -> Optional[np.ndarray]:
if self.render_mode == "rgb_array":
return self.state.copy()
print(f"rendered: {self.state} | {self.desired_goal}")
if __name__ == "__main__":
eval_callback = EvalCallback(
Monitor(ClearerRenderEnv(n_bits=2)),
n_eval_episodes=1,
eval_freq=10_000,
render=True,
)
environment = make_vec_env(
BitFlippingEnv,
env_kwargs={"n_bits": 2},
n_envs=1
)
model = DQN(
"MultiInputPolicy",
environment,
learning_starts=0,
learning_rate=0.1,
verbose=2
)
model.learn(
total_timesteps=10_000,
log_interval=None,
callback=eval_callback,
progress_bar=True,
)Relevant log output / Error message
Notice the first state [0, 0] is never rendered and neither is the last step, only the reset frame (in this case, also [0, 0])
step : [0 0] -> [0 1] | [1 1]
rendered: [0 1] | [1 1]
step : [0 1] -> [1 1] | [1 1]
reset : [1 1] -> [0 0] | [1 1]
rendered: [0 0] | [1 1]
Eval num_timesteps=10000, episode_reward=-1.00 +/- 0.00
Episode length: 2.00 +/- 0.00
Success rate: 100.00%For a new user only looking at the rendered messages, this would be very confusing. Here's another example where it seems like the agent changed two values at once.
step : [1 1] -> [1 0] | [1 1]
rendered: [1 0] | [1 1]
step : [1 0] -> [1 1] | [1 1]
reset : [1 1] -> [0 1] | [1 1]
rendered: [0 1] | [1 1]
Eval num_timesteps=10, episode_reward=-1.00 +/- 0.00
Episode length: 2.00 +/- 0.00
Success rate: 100.00%System Info
No response
Checklist
- My issue does not relate to a custom gym environment. (Use the custom gym env template instead)
- I have checked that there is no similar issue in the repo
- I have read the documentation
- I have provided a minimal and working example to reproduce the bug
- I've used the markdown code blocks for both code and stack traces.
Source: DLR-RM/stable-baselines3