PPO Continues action
I tried to run ppo_continues_action.py at different seeds from 0 to 9. I got totally different results. Some models achieved high episodic returns during training, but performed very poorly during evaluation—even scoring negatively. I only changed the random seed in the entire code. I hope somebody help me to figure out what's going on. I’ve included my results and pip list below.
gymnasium: 0.29.1 mujoco：3.3.7
parameter：
class Args: exp_name: str = os.path.basename(__file__)[: -len(".py")] """the name of this experiment""" seed: int = 0 """seed of the experiment""" torch_deterministic: bool = True """if toggled, torch.backends.cudnn.deterministic=False""" cuda: bool = True """if toggled, cuda will be enabled by default""" track: bool = True """if toggled, this experiment will be tracked with Weights and Biases""" wandb_project_name: str = "cleanRL" """the wandb's project name""" wandb_entity: str = 'bilibili' """the entity (team) of wandb's project""" capture_video: bool = False """whether to capture videos of the agent performances (check out videosfolder)""" save_model: bool = True """whether to save model into theruns/{run_name}` folder"""
upload_model: bool = False
"""whether to upload the saved model to huggingface"""
hf_entity: str = ""
"""the user or org name of the model repository from the Hugging Face Hub"""
# Algorithm specific arguments
env_id: str = "HalfCheetah-v4"
"""the id of the environment"""
total_timesteps: int = 1000000
"""total timesteps of the experiments"""
learning_rate: float = 3e-4
"""the learning rate of the optimizer"""
num_envs: int = 1
"""the number of parallel game environments"""
num_steps: int = 2048
"""the number of steps to run in each environment per policy rollout"""
anneal_lr: bool = True
"""Toggle learning rate annealing for policy and value networks"""
gamma: float = 0.99
"""the discount factor gamma"""
gae_lambda: float = 0.95
"""the lambda for the general advantage estimation"""
num_minibatches: int = 32
"""the number of mini-batches"""
update_epochs: int = 10
"""the K epochs to update the policy"""
norm_adv: bool = True
"""Toggles advantages normalization"""
clip_coef: float = 0.2
"""the surrogate clipping coefficient"""
clip_vloss: bool = True
"""Toggles whether or not to use a clipped loss for the value function, as per the paper."""
ent_coef: float = 0.0
"""coefficient of the entropy"""
vf_coef: float = 0.5
"""coefficient of the value function"""
max_grad_norm: float = 0.5
"""the maximum norm for the gradient clipping"""
target_kl: float = None
"""the target KL divergence threshold"""
# to be filled in runtime
batch_size: int = 0
"""the batch size (computed in runtime)"""
minibatch_size: int = 0
"""the mini-batch size (computed in runtime)"""
num_iterations: int = 0
"""the number of iterations (computed in runtime)"""`
result：
Source: vwxyzjn/cleanrl