Multi-GPU gives no speedup for transformer model

Description

I am training a Transformer model on the Librispeech dataset using 4 GPUs with 8 CPU-cores.

I have tested the following:

Single-GPU

export CUDA_VISIBLE_DEVICES=0

t2t-trainer \
  --worker-gpu=1 \
  # ..

Multi-GPU

export CUDA_VISIBLE_DEVICES=0,1,2,3

t2t-trainer \
  --worker-gpu=4 \
  # ..

Both scripts are working. The training starts and on the surface everything looks okay. However, I am getting a global_step/sec or just ~2 steps for Multi-GPU, compared to ~9 steps for Single-GPU.

Shouldn't I see a speedup using multiple GPUs? If so: What might be the problem here? Can I trust the log output?

Environment information

OS: Linux #37~16.04.1-Ubuntu SMP Tue Aug 28 10:44:06 UTC 2018 GNU/Linux

$ pip freeze | grep tensor
tensor2tensor==1.9.0
tensorboard==1.10.0
tensorflow-gpu==1.10.1

$ python -V
Python 3.5.6 :: Anaconda, Inc.