Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Tensor2Tensor, or T2T for short, is a library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
T2T was developed by researchers and engineers in the Google Brain team and a community of users. It is now deprecated — we keep it running and welcome bug-fixes, but encourage users to use the successor library Trax.
This iPython notebook explains T2T and runs in your browser using a free VM from Google, no installation needed. Alternatively, here is a one-command version that installs T2T, downloads MNIST, trains a model and evaluates it:
pip install tensor2tensor && t2t-trainer \
--generate_data \
--data_dir=~/t2t_data \
--output_dir=~/t2t_train/mnist \
--problem=image_mnist \
--model=shake_shake \
--hparams_set=shake_shake_quick \
--train_steps=1000 \
--eval_steps=100
Below we list a number of tasks that can be solved with T2T when you train the appropriate model on the appropriate problem. We give the problem and model below and we suggest a setting of hyperparameters that we know works well in our setup. We usually run either on Cloud TPUs or on 8-GPU machines; you might need to modify the hyperparameters if you run on a different setup.
For evaluating mathematical expressions at the character level involving addition, subtraction and multiplication of both positive and negative decimal numbers with variable digits assigned to symbolic variables, use
--problem=algorithmic_math_two_variablesYou can try solving the problem with different transformer models and hyperparameters as described in the paper:
--model=transformer
--hparams_set=transformer_tiny--model=universal_transformer
--hparams_set=universal_transformer_tiny--model=universal_transformer
--hparams_set=adaptive_universal_transformer_tinyFor answering questions based on a story, use
--problem=babi_qa_concat_task1_1kYou can choose the bAbi task from the range [1,20] and the subset from 1k or
10k. To combine test data from all tasks into a single test set, use
--problem=babi_qa_concat_all_tasks_10k
For image classification, we have a number of standard data-sets:
--problem=image_imagenet, or one
of the re-scaled versions (image_imagenet224, image_imagenet64,
image_imagenet32)--problem=image_cifar10 (or
--problem=image_cifar10_plain to turn off data augmentation)--problem=image_cifar100--problem=image_mnistFor ImageNet, we suggest to use the ResNet or Xception, i.e.,
use --model=resnet --hparams_set=resnet_50 or
--model=xception --hparams_set=xception_base.
Resnet should get to above 76% top-1 accuracy on ImageNet.
For CIFAR and MNIST, we suggest to try the shake-shake model:
--model=shake_shake --hparams_set=shakeshake_big.
This setting trained for --train_steps=700000 should yield
close to 97% accuracy on CIFAR-10.
For (un)conditional image generation, we have a number of standard data-sets:
--problem=img2img_celeba for image-to-image translation, namely,
superresolution from 8x8 to 32x32.--problem=image_celeba256_rev for a downsampled 256x256.--problem=image_cifar10_plain_gen_rev for class-conditional
32x32 generation.--problem=image_lsun_bedrooms_rev--problem=image_text_ms_coco_rev for text-to-image generation.--problem=image_imagenet32_gen_rev for
32x32 or --problem=image_imagenet64_gen_rev for 64x64.We suggest to use the Image Transformer, i.e., --model=imagetransformer, or
the Image Transformer Plus, i.e., --model=imagetransformerpp that uses
discretized mixture of logistics, or variational auto-encoder, i.e.,
--model=transformer_ae.
For CIFAR-10, using --hparams_set=imagetransformer_cifar10_base or
--hparams_set=imagetransformer_cifar10_base_dmol yields 2.90 bits per
dimension. For Imagenet-32, using
--hparams_set=imagetransformer_imagenet32_base yields 3.77 bits per dimension.
For language modeling, we have these data-sets in T2T:
--problem=languagemodel_ptb10k for
word-level modeling and --problem=languagemodel_ptb_characters
for character-level modeling.--problem=languagemodel_lm1b32k for
subword-level modeling and --problem=languagemodel_lm1b_characters
for character-level modeling.We suggest to start with --model=transformer on this task and use
--hparams_set=transformer_small for PTB and
--hparams_set=transformer_base for LM1B.
For the task of recognizing the sentiment of a sentence, use
--problem=sentiment_imdbWe suggest to use --model=transformer_encoder here and since it is
a small data-set, try --hparams_set=transformer_tiny and train for
few steps (e.g., --train_steps=2000).
For speech-to-text, we have these data-sets in T2T:
Librispeech (US English): --problem=librispeech for
the whole set and --problem=librispeech_clean for a smaller
but nicely filtered part.
Mozilla Common Voice (US English): --problem=common_voice for the whole set
--problem=common_voice_clean for a quality-checked subset.
For summarizing longer text into shorter one we have these data-sets:
--problem=summarize_cnn_dailymail32kWe suggest to use --model=transformer and
--hparams_set=transformer_prepend for this task.
This yields good ROUGE scores.
There are a number of translation data-sets in T2T:
--problem=translate_ende_wmt32k--problem=translate_enfr_wmt32k--problem=translate_encs_wmt32k--problem=translate_enzh_wmt32k--problem=translate_envi_iwslt32k--problem=translate_enes_wmt32kYou can get translations in the other direction by appending _rev to
the problem name, e.g., for German-English use
--problem=translate_ende_wmt32k_rev
(note that you still need to download the original data with t2t-datagen
--problem=translate_ende_wmt32k).
For all translation problems, we suggest to try the Transformer model:
--model=transformer. At first it is best to try the base setting,
--hparams_set=transformer_base. When trained on 8 GPUs for 300K steps
this should reach a BLEU score of about 28 on the English-German data-set,
which is close to state-of-the art. If training on a single GPU, try the
--hparams_set=transformer_base_single_gpu setting. For very good results
or larger data-sets (e.g., for English-French), try the big model
with --hparams_set=transformer_big.
See this example to know how the translation works.
Here's a walkthrough training a good English-to-German translation model using the Transformer model from Attention Is All You Need on WMT data.
…
# Assumes tensorflow or tensorflow-gpu installed
pip install tensor2tensor
# Installs with tensorflow-gpu requirement
pip install tensor2tensor[tensorflow_gpu]
# Installs with tensorflow (cpu) requirement
pip install tensor2tensor[tensorflow]
Binaries:
# Data generator
t2t-datagen
# Trainer
t2t-trainer --registry_help
Library usage:
python -c "from tensor2tensor.models.transformer import Transformer"
bottom and top transformations, which are specified per-feature in the
model.t2t-datagen and the training script t2t-trainer.Problems consist of features such as inputs and targets, and metadata such
as each feature's modality (e.g. symbol, image, audio) and vocabularies. Problem
features are given by a dataset, which is stored as a TFRecord file with
tensorflow.Example protocol buffers. All
problems are imported in
all_problems.py
or are registered with @registry.register_problem. Run
t2t-datagen
to see the list of available problems and download them.
T2TModels define the core tensor-to-tensor computation. They apply a
default transformation to each input and output so that models may deal with
modality-independent tensors (e.g. embeddings at the input; and a linear
transform at the output to produce logits for a softmax over classes). All
models are imported in the
models subpackage,
inherit from T2TModel,
and are registered with
[@registry.register_model