ignite

TL;DR

Ignite is a high-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

Features

• Less code than pure PyTorch while ensuring maximum control and simplicity

• Library approach and no program's control inversion - Use ignite where and when you need

• Extensible API for metrics, experiment managers, and other components

Table of Contents

• Table of Contents
• Why Ignite?
  • Simplified training and validation loop
  • Power of Events & Handlers
    • Execute any number of functions whenever you wish
    • Built-in events filtering
    • Stack events to share some actions
    • Custom events to go beyond standard events
  • Out-of-the-box metrics
• Installation
  • Nightly releases
  • Docker Images
    • Using pre-built images
• Getting Started
• Documentation
  • Additional Materials
• Examples
  • Tutorials
  • Reproducible Training Examples
• Communication
  • User feedback
• Contributing
• Projects using Ignite
• Citing Ignite
• About the team & Disclaimer

Why Ignite?

Ignite is a library that provides three high-level features:

• Extremely simple engine and event system
• Out-of-the-box metrics to easily evaluate models
• Built-in handlers to compose training pipeline, save artifacts and log parameters and metrics

Simplified training and validation loop

No more coding for/while loops on epochs and iterations. Users instantiate engines and run them.

Example

…

Power of Events & Handlers

The cool thing with handlers is that they offer unparalleled flexibility (compared to, for example, callbacks). Handlers can be any function: e.g. lambda, simple function, class method, etc. Thus, we do not require to inherit from an interface and override its abstract methods which could unnecessarily bulk up your code and its complexity.

Execute any number of functions whenever you wish

Examples

…

Built-in events filtering

Examples

# run the validation every 5 epochs
@trainer.on(Events.EPOCH_COMPLETED(every=5))
def run_validation():
    # run validation

# change some training variable once on 20th epoch
@trainer.on(Events.EPOCH_STARTED(once=20))
def change_training_variable():
    # ...

# Trigger handler with customly defined frequency
@trainer.on(Events.ITERATION_COMPLETED(event_filter=first_x_iters))
def log_gradients():
    # ...

Stack events to share some actions

Examples

Events can be stacked together to enable multiple calls:

@trainer.on(Events.COMPLETED | Events.EPOCH_COMPLETED(every=10))
def run_validation():
    # ...

Custom events to go beyond standard events

Examples

Custom events related to backward and optimizer step calls:

…

• Complete snippet is found here.
• Another use-case of custom events: trainer for Truncated Backprop Through Time.

Out-of-the-box metrics

• Metrics for various tasks: Precision, Recall, Accuracy, Confusion Matrix, IoU etc, ~20 regression metrics.

• Users can also compose their metrics with ease from existing ones using arithmetic operations or torch methods.

Example

precision = Precision(average=False)
recall = Recall(average=False)
F1_per_class = (precision * recall * 2 / (precision + recall))
F1_mean = F1_per_class.mean()  # torch mean method
F1_mean.attach(engine, "F1")

Installation

From pip:

pip install pytorch-ignite

From conda:

conda install ignite -c pytorch

From source:

pip install git+https://github.com/pytorch/ignite

Nightly releases

From pip:

pip install --pre pytorch-ignite

From conda (this suggests to install pytorch nightly release instead of stable version as dependency):

conda install ignite -c pytorch-nightly

Docker Images

Using pre-built images

Pull a pre-built docker image from our Docker Hub and run it with docker v19.03+.

docker run --gpus all -it -v $PWD:/workspace/project --network=host --shm-size 16G pytorchignite/base:latest /bin/bash

List of available pre-built images

Base

• pytorchignite/base:latest
• pytorchignite/apex:latest
• pytorchignite/hvd-base:latest
• pytorchignite/hvd-apex:latest
• pytorchignite/msdp-apex:latest

Vision:

• pytorchignite/vision:latest
• pytorchignite/hvd-vision:latest
• pytorchignite/apex-vision:latest
• pytorchignite/hvd-apex-vision:latest
• pytorchignite/msdp-apex-vision:latest

NLP:

• pytorchignite/nlp:latest
• pytorchignite/hvd-nlp:latest
• pytorchignite/apex-nlp:latest
• pytorchignite/hvd-apex-nlp:latest
• pytorchignite/msdp-apex-nlp:latest

For more details, see here.

Getting Started

Few pointers to get you started:

• Quick Start Guide: Essentials of getting a project up and running
• Concepts of the library: Engine, Events & Handlers, State, Metrics
• Full-featured template examples (coming soon)

Documentation

• Stable API documentation and an overview of the library: https://pytorch.org/ignite/
• Development version API documentation: https://pytorch.org/ignite/master/
• FAQ, "Questions on Github" and "Questions on Discuss.PyTorch".
• Project's Roadmap

Additional Materials

• Distributed Training Made Easy with PyTorch-Ignite
• PyTorch Ecosystem Day 2021 Breakout session presentation
• Tutorial blog post about PyTorch-Ignite
• 8 Creators and Core Contributors Talk About Their Model Training Libraries From PyTorch Ecosystem
• Ignite Posters from Pytorch Developer Conferences:
  • 2021
  • 2019
  • 2018

Examples

Tutorials

• Text Classification using Convolutional Neural Networks
• Variational Auto Encoders
• Convolutional Neural Networks for Classifying Fashion-MNIST Dataset
• Training Cycle-GAN on Horses to Zebras with Nvidia/Apex - logs on W&B
• Another training Cycle-GAN on Horses to Zebras with Native Torch CUDA AMP - logs on W&B
• Finetuning EfficientNet-B0 on CIFAR100
• Hyperparameters tuning with Ax
• Basic example of LR finder on MNIST
• Benchmark mixed precision training on Cifar100: torch.amp vs nvidia/apex
• MNIST training on a single TPU
• CIFAR10 Training on multiple TPUs
• Basic example of handlers time profiling on MNIST training example

Reproducible Training Examples

Inspired by torchvision/references, we provide several reproducible baselines for vision tasks:

• ImageNet - logs on Ignite Trains server coming soon ...
• Pascal VOC2012 - logs on Ignite Trains server coming soon ...

Features:

• Distributed training: native or horovod and using PyTorch native AMP

Code-Generator application

The easiest way to create your training scripts with PyTorch-Ignite:

• https://c