Do you need to set `retain_graph=True`?

Hi,

I think you should use retain_graph=True here, because otherwise, the graph will be freed from there and update of value function will not affect the update of the actor.