#1920·tensor2tensor

Aadm is slower than Adafactor

Author: shizhediaoCreated Nov 26, 2022Updated Nov 26, 2022

Hi, I found that training Transformers with Adam is three times slower than with Adafactor. Here is the command I am using for Adam:

bash

t2t-trainer \
  --data_dir=./t2t/t2t_data \
  --problem=translate_ende_wmt32k \
  --model=transformer \
  --hparams_set=transformer_base \
  --hparams="batch_size=1024,learning_rate_schedule=constant*linear_warmup*rsqrt_decay, learning_rate_constant=0.1,optimizer_adam_beta2=0.999" \
  --schedule=continuous_train_and_eval \
  --output_dir=./t2t/t2t_train/translate_ende_wmt32k_adam_lineB \
  --train_steps=300000 \
  --worker_gpu=10 \
  --eval_steps=5000

Here is the command I am using for Adafactor:

bash

t2t-trainer \
  --data_dir=./t2t/t2t_data \
  --problem=translate_ende_wmt32k \
  --model=transformer \
  --hparams_set=transformer_base \
  --hparams="optimizer_adafactor_factored=False,batch_size=1024,optimizer=Adafactor,learning_rate_schedule=constant*linear_warmup*rsqrt_decay, learning_rate_constant=0.1,optimizer_adafactor_multiply_by_parameter_scale=False" \
  --schedule=continuous_train_and_eval \
  --output_dir=./t2t/t2t_train/translate_ende_wmt32k_adafactor_lineN \
  --train_steps=300000 \
  --worker_gpu=10 \
  --eval_steps=5000

I found that training 100 steps cost 240 seconds for Adam, while it just needs 80s for Adafactor. Could anyone help take a look?

Thanks very much!

Source: tensorflow/tensor2tensor

View original on GitHub View discussion on GitHub