pytorch-kaldi

The PyTorch-Kaldi Speech Recognition Toolkit

PyTorch-Kaldi is an open-source repository for developing state-of-the-art DNN/HMM speech recognition systems. The DNN part is managed by PyTorch, while feature extraction, label computation, and decoding are performed with the Kaldi toolkit.

This repository contains the last version of the PyTorch-Kaldi toolkit (PyTorch-Kaldi-v1.0). To take a look into the previous version (PyTorch-Kaldi-v0.1), click here.

If you use this code or part of it, please cite the following paper:

M. Ravanelli, T. Parcollet, Y. Bengio, "The PyTorch-Kaldi Speech Recognition Toolkit", arXiv

@inproceedings{pytorch-kaldi,
title    = {The PyTorch-Kaldi Speech Recognition Toolkit},
author    = {M. Ravanelli and T. Parcollet and Y. Bengio},
booktitle    = {In Proc. of ICASSP},
year    = {2019}
}

The toolkit is released under a Creative Commons Attribution 4.0 International license. You can copy, distribute, modify the code for research, commercial and non-commercial purposes. We only ask to cite our paper referenced above.

To improve transparency and replicability of speech recognition results, we give users the possibility to release their PyTorch-Kaldi model within this repository. Feel free to contact us (or doing a pull request) for that. Moreover, if your paper uses PyTorch-Kaldi, it is also possible to advertise it in this repository.

See a short introductory video on the PyTorch-Kaldi Toolkit

SpeechBrain

We are happy to announce that the SpeechBrain project (https://speechbrain.github.io/) is now public! We strongly encourage users to migrate to Speechbrain. It is a much better project which already supports several speech processing tasks, such as speech recognition, speaker recognition, SLU, speech enhancement, speech separation, multi-microphone signal processing and many others.

The goal is to develop a single, flexible, and user-friendly toolkit that can be used to easily develop state-of-the-art speech systems for speech recognition (both end-to-end and HMM-DNN), speaker recognition, speech separation, multi-microphone signal processing (e.g, beamforming), self-supervised learning, and many others.

The project will be lead by Mila and is sponsored by Samsung, Nvidia, Dolby. SpeechBrain will also benefit from the collaboration and expertise of other companies such as Facebook/PyTorch, IBMResearch, FluentAI.

We are actively looking for collaborators. Feel free to contact us at [email protected] if you are interested to collaborate.

Thanks to our sponsors we are also able to hire interns working at Mila on the SpeechBrain project. The ideal candidate is a PhD student with experience on pytorch and speech technologies (send your CV to [email protected])

The development of SpeechBrain will require some months before having a working repository. Meanwhile, we will continue providing support for the pytorch-kaldi project.

Stay Tuned!

Table of Contents

• Introduction
• Prerequisites
• How to install
• Recent Updates
• Tutorials:
  • TIMIT tutorial
  • Librispeech tutorial
• Toolkit Overview:
  • Toolkit architecture
  • Configuration files
• FAQs:
  • How can I plug-in my model?
  • How can I tune the hyperparameters?
  • How can I use my own dataset?
  • How can I plug-in my own features?
  • How can I transcript my own audio files?
  • Batch size, learning rate, and droput scheduler
  • How can I contribute to the project?
• EXTRA:
  • Speech recognition from the raw waveform with SincNet
  • Joint training between speech enhancement and ASR
  • Distant Speech Recognition with DIRHA
  • Training an autoencoder
• References

Introduction

The PyTorch-Kaldi project aims to bridge the gap between the Kaldi and the PyTorch toolkits, trying to inherit the efficiency of Kaldi and the flexibility of PyTorch. PyTorch-Kaldi is not only a simple interface between these toolkits, but it embeds several useful features for developing modern speech recognizers. For instance, the code is specifically designed to naturally plug-in user-defined acoustic models. As an alternative, users can exploit several pre-implemented neural networks that can be customized using intuitive configuration files. PyTorch-Kaldi supports multiple feature and label streams as well as combinations of neural networks, enabling the use of complex neural architectures. The toolkit is publicly-released along with rich documentation and is designed to properly work locally or on HPC clusters.

Some features of the new version of the PyTorch-Kaldi toolkit:

• Easy interface with Kaldi.
• Easy plug-in of user-defined models.
• Several pre-implemented models (MLP, CNN, RNN, LSTM, GRU, Li-GRU, SincNet).
• Natural implementation of complex models based on multiple features, labels, and neural architectures.
• Easy and flexible configuration files.
• Automatic recovery from the last processed chunk.
• Automatic chunking and context expansions of the input features.
• Multi-GPU training.
• Designed to work locally or on HPC clusters.
• Tutorials on TIMIT and Librispeech Datasets.

Prerequisites

1. If not already done, install Kaldi (http://kaldi-asr.org/). As suggested during the installation, do not forget to add the path of the Kaldi binaries into $HOME/.bashrc. For instance, make sure that .bashrc contains the following paths:

export KALDI_ROOT=/home/mirco/kaldi-trunk
PATH=$PATH:$KALDI_ROOT/tools/openfst
PATH=$PATH:$KALDI_ROOT/src/featbin
PATH=$PATH:$KALDI_ROOT/src/gmmbin
PATH=$PATH:$KALDI_ROOT/src/bin
PATH=$PATH:$KALDI_ROOT//src/nnetbin
export PATH

Remember to change the KALDI_ROOT variable using your path. As a first test to check the installation, open a bash shell, type "copy-feats" or "hmm-info" and make sure no errors appear.

1. If not already done, install PyTorch (http://pytorch.org/). We tested our codes on PyTorch 1.0 and PyTorch 0.4. An older version of PyTorch is likely to raise errors. To check your installation, type “python” and, once entered into the console, type “import torch”, and make sure no errors appear.

2. We recommend running the code on a GPU machine. Make sure that the CUDA libraries (https://developer.nvidia.com/cuda-downloads) are installed and correctly working. We tested our system on Cuda 9.0, 9.1 and 8.0. Make sure that python is installed (the code is tested with python 2.7 and python 3.7). Even though not mandatory, we suggest using Anaconda (https://anaconda.org/anaconda/python).

Recent updates

19 Feb. 2019: updates:

• It is now possible to dynamically change batch size, learning rate, and dropout factors during training. We thus implemented a scheduler that supports the following formalism within the config files:

batch_size_train = 128*12 | 64*10 | 32*2

The line above means: do 12 epochs with 128 batches, 10 epochs with 64 batches, and 2 epochs with 32 batches. A similar formalism can be used for learning rate and dropout scheduling. See this section for more information.

5 Feb. 2019: updates:

1. Our toolkit now supports parallel data loading (i.e., the next chunk is stored in memory while processing the current chunk). This allows a significant speed up.
2. When performing monophone regularization users can now set “dnn_lay = N_lab_out_mono”. This way the number of monophones is automatically inferred by our toolkit.
3. We integrated the kaldi-io toolkit from the kaldi-io-for-python project into data_io-py.
4. We provided a better hyperparameter setting for SincNet (see this section)
5. We released some baselines with the DIRHA dataset (see this section). We also provide some configuration examples for a simple autoencoder (see this section) and for a system that jointly trains a speech enhancement and a speech recognition module (see this section)
6. We fixed some minor bugs.

Notes on the next version: In the next version, we plan to further extend the functionalities of our toolkit, supporting more models and features formats. The goal is to make our toolkit suitable for other speech-related tasks such as end-to-end speech recognition, speaker-identification, keyword spotting, speech separation, speech activity detection, speech enhancement, etc. If you would like to propose some novel functionalities, please give us your feedback by filling this survey.

How to install

To install PyTorch-Kaldi, do the following steps:

1. Make sure all the software recommended in the “Prerequisites” sections are installed and are correctly working
2. Clone the PyTorch-Kaldi repository:

git clone https://github.com/mravanelli/pytorch-kaldi

1. Go into the project folder and Install the needed packages with:

pip install -r requirements.txt

TIMIT tutorial

In the following, we provide a short tutorial of the PyTorch-Kaldi toolkit based on the popular TIMIT dataset.

1. Make sure you have the TIMIT dataset. If not, it can be downloaded from the LDC website (https://catalog.ldc.upenn.edu/LDC93S1).

2. Make sure Kaldi and PyTorch installations are fine. Make also sure that your KALDI paths are currently working (you should add the Kaldi paths into the .bashrc as reported in the section "Prerequisites"). For instance, type "copy-feats" and "hmm-info" and make sure no errors appear.

3. Run the Kaldi s5 baseline of TIMIT. This step is necessary to compute features and labels later used to train the PyTorch neural network. We recommend running the full timit s5 recipe (including the DNN training):

cd kaldi/egs/timit/s5
./run.sh
./local/nnet/run_dnn.sh

This way all the necessary files are created and the user can directly compare the results obtained by Kaldi with that achieved with our toolkit.

1. Compute the alignments (i.e, the phone-state labels) for test and dev data with the following commands (go into $KALDI_ROOT/egs/timit/s5). If you want to use tri3 alignments, type:

steps/align_fmllr.sh --nj 4 data/dev data/lang exp/tri3 exp/tri3_ali_dev

steps/align_fmllr.sh --nj 4 data/test data/lang exp/tri3 exp/tri3_ali_test

If you want to use dnn alignments (as suggested), type:

steps/nnet/align.sh --nj 4 data-fmllr-tri3/train data/lang exp/dnn4_pretrain-dbn_dnn exp/dnn4_pretrain-dbn_dnn_ali

steps/nnet/align.sh --nj 4 data-fmllr-tri3/dev data/lang exp/dnn4_pretrain-dbn_dnn exp/dnn4_pretrain-dbn_dnn_ali_dev

steps/nnet/align.sh --nj 4 data-fmllr-tri3/test data/lang exp/dnn4_pretrain-dbn_dnn exp/dnn4_pretrain-dbn_dnn_ali_test

1. We start this tutorial with a very simple MLP network trained on mfcc features. Before launching the experiment, take a lo