#248·sonnet

Can you explain the logic of updated_ema_cluster_size in VectorQuantizerEMA?

Author: wztdreamCreated Jul 21, 2022Updated Jul 21, 2022

Hi, It seems updated_ema_cluster_size in VectorQuantizerEMA use small epsilon to avoid zero in denominator, but the formula is strange to me, so can you explain the logic here, and why not directly use something like x=x+epsilon? https://github.com/deepmind/sonnet/blob/d1cd37117bcb98223b3e4b930717d418abb76484/sonnet/src/nets/vqvae.py#L270-L271