Stuck after printing 'Successfully opened dynamic library libcublas.so.10.0'

Author: zhez6Created Jul 24, 2019Updated Mar 22, 2023

Description

I run this command 't2t-trainer --problem=librispeech --model=transformer --data_dir=~/dataset/t2t/librispeech/ --output_dir=. --hparams_set=transformer_librispeech --worker_gpu=1' and it's stuck after printing 'Successfully opened dynamic library libcublas.so.10.0'. Then I set TF_CPP_MIN_VLOG_LEVEL=2, it keeps printing 'tensorflow/core/framework/model.cc:440] Starting optimization of tunable parameters tensorflow/core/framework/model.cc:482] Number of tunable parameters: 0 tensorflow/core/kernels/data/model_dataset_op.cc:172] Waiting for 20480 ms. tensorflow/core/framework/model.cc:440] Starting optimization of tunable parameters tensorflow/core/framework/model.cc:482] Number of tunable parameters: 0 tensorflow/core/kernels/data/model_dataset_op.cc:172] Waiting for 40960 ms. tensorflow/core/framework/model.cc:440] Starting optimization of tunable parameters tensorflow/core/framework/model.cc:482] Number of tunable parameters: 0 tensorflow/core/kernels/data/model_dataset_op.cc:172] Waiting for 60000 ms. tensorflow/core/framework/model.cc:440] Starting optimization of tunable parameters tensorflow/core/framework/model.cc:482] Number of tunable parameters: 0 tensorflow/core/kernels/data/model_dataset_op.cc:172] Waiting for 60000 ms.'

...

Environment information

OS: <your answer here>

$ pip freeze | grep tensor
# your output here
mesh-tensorflow==0.0.5
tensor2tensor==1.13.4
tensorboard==1.14.0
tensorflow-datasets==1.0.2
tensorflow-estimator==1.14.0
tensorflow-gpu==1.14.0
tensorflow-metadata==0.14.0
tensorflow-probability==0.7.0
$ python -V
# your output here
Python 3.6.5 :: Anaconda, Inc.

For bugs: reproduction and error logs

# Steps to reproduce:
# Error logs:
See descriptions

Source: tensorflow/tensor2tensor