How to make livetalking allow more sessions?

I am currently testing multi sessions with livetalking on an RTX 3090, but I noticed that when working with multi sessions (especiall with musetalk), the gpu memory gets filled to the half the moment it starts working (talking) and does not easily free up. When attemting to open more sessions and use them in parallel, As soon as I go beyond 2 sessions and make them talk in parallel, it starts lagging, stuttering and at some point completely freezes with a gpu working at 100% and a VRAM still available (almost 40% free).

I am really confused with this behavior and I am not sure how to work on it to fix and make the system use the resources more efficiently. I saw some posts about the use of processes vs threads explaining it was one of the causes. is there any way to work around this or will there be any update in the code soon to make this process more efficient?

Thank you.