QNNPACK

QNNPACK

QNNPACK (Quantized Neural Networks PACKage) is a mobile-optimized library for low-precision high-performance neural network inference. QNNPACK provides implementation of common neural network operators on quantized 8-bit tensors.

QNNPACK is not intended to be directly used by machine learning researchers; instead it provides low-level performance primitives for high-level deep learning frameworks. As of today, QNNPACK is integrated in PyTorch 1.0 with Caffe2 graph representation.

Operator Coverage

Currently implemented and planned for implementation operators are below:

• 2D Convolution
• 2D Deconvolution
• Channel Shuffle
• Fully Connected
• Locally Connected
• 2D Max Pooling
• 2D Average Pooling
• Global Average Pooling
• Sigmoid
• Leaky ReLU
• Clamp (can be used for ReLU, ReLU6 if it is not fused in another operator)
• SoftArgMax (aka SoftMax)
• Group Normalization

Building

QNNPACK provides standard CMake-based build scripts.

Native compilation

Users are recommended to use scripts/build-local.sh script to build QNNPACK for the host machine.

Cross-compilation for Android

To cross-compile for Android, set $ANDROID_NDK environment variable (where $ANDROID_NDK is the path to Android NDK directory, e.g. /opt/android-ndk-r15c) and use one of the scripts from the table below:

ABI	Build script	Restrictions
armeabi-v7a	scripts/build-android-armv7.sh	Requires CPU with ARM NEON
arm64-v8a	scripts/build-android-arm64.sh
x86	scripts/build-android-x86.sh

Notes:

• On armeabi-v7a qnnp_initialize will fail with qnnp_status_unsupported_hardware if the mobile CPU does not support ARM NEON. Don't set -DANDROID_ARM_NEON=1 for QNNPACK compilation as it can make qnnp_initialize crash on CPUs without ARM NEON.

Cross-compilation for iOS

To cross-compile for iOS, clone ios-cmake, and set $IOS_CMAKE_TOOLCHAIN_FILE environment variable (where $IOS_CMAKE_TOOLCHAIN_FILE is the path to ios.toolchain.cmake file in ios-cmake), and use one of the scripts from the table below:

Architecture	Build script	Notes
armv7	scripts/build-ios-armv7.sh	iPhone 3GS/4/4S
armv7	scripts/build-ios-armv7s.sh	iPhone 5 and newer
arm64	scripts/build-ios-arm64.sh	iPhone 5S and newer
arm64e	scripts/build-ios-arm64e.sh	iPhone XS/XR
i386	scripts/build-ios-i386.sh	iPhone Simulator (32-bit)
x86_64	scripts/build-ios-x86_64.sh	iPhone Simulator (64-bit)

End-to-End Benchmarking

Caffe2 backend of PyTorch 1.0 natively integrates QNNPACK, and provides a pre-trained quantized MobileNet v2 model. Below are instructions for benchmarking this model end-to-end with QNNPACK.

Raspberry Pi 2 or 3

…

ARMv7 (32-bit) Android

…

ARM64 (64-bit) Android

…

PEP (Performance Evaluation Platform) Method

Facebook AI Performance Evaluation Platform is a framework and backend agnostic benchmarking platform to compare machine learning inferencing runtime metrics on a set of models and a variety of backends.

We use PEP to produce the results we have in our blog

With an ARMv7 device connected:

…

Acknowledgements

QNNPACK is developed by Marat Dukhan, Yiming Wu, Hao Lu, and Bert Maher. We thank Andrew Tulloch and Yangqing Jia for advice during the development of QNNPACK.

License

QNNPACK is BSD licensed, as found in the LICENSE file.