Question about Classifier-free Guidance

Author: lucalaCreated Apr 15, 2023Updated Jan 8, 2024

When embedding my text for conditioning, the trick for classifier-free guidance is to drop the embedding sometimes (usually 10% of the time).

My question is, what does drop mean? It seems I have come across two variants: using a random tensor as a substitute or a zero tensor.

GLIDE mentions in section 2.3 "we sometimes replace text captions with an empty sequence" - this would be a third option, using the embedding from the empty string?

I haven't been able to find any explanation on this, does someone know?

Source: lucidrains/imagen-pytorch