Low Loss, Bad Results

Greetings. I'll preface my question with a disclaimer that I don't have much experience in ML and I'm still exploring myself, so I apologize if this question may sound silly or too general.

I'm using this architecture and library to train a model of my own on a certain type of latents. If I set the training goal to predict the noise at each step my model successfully reaches low loss values (~0.15). Yet the samples it produces are nothing like the original. Only setting the goal to predict the original x latent works. I don't understand why that is, maybe you can at least give me an idea to a potential cause or give me an intuition for the problem?

Any help would be very appreciated, thank you.