libxsmm

Open source

Library for specialized dense and sparse matrix operations, and deep learning primitives.

970 stars0 likes1 views

About

Library for specialized dense and sparse matrix operations, and deep learning primitives.

LIBXSMM

LIBXSMM is high performance library for small dense and sparse linear algebra opertions incl. GEMM and elementwise primities often seen in deep learning applications. It also serves as reference implementation of Tensor Processing Primitives (TPP), a programming abstraction for efficient and portable deep learning and HPC workloads. With version 2.0, LIBXSMM focuses on providing a complete and architecture-portable set of TPPs (small dense and sparse matrix operations as well as element-wise, GEMM, and BRGEMM primitives) from which higher-level operators such as convolutions, fully-connected layers, normalization, and pooling are composed. LIBXSMM targets Intel Architecture with Intel SSE, Intel AVX, Intel AVX2, Intel AVX‑512 (with VNNI and Bfloat16), and Intel AMX (Advanced Matrix Extensions), AArch64 (NEON, SVE, and SME), RISC‑V (RVV), and PowerPC 64‑bit little‑endian (POWER10 with VSX and MMA). Code generation is mainly based on Just‑In‑Time (JIT) code specialization for compiler-independent performance (matrix multiplications, matrix transpose/copy, sparse functionality, and tensor primitives). LIBXSMM is suitable for "build once and deploy everywhere", i.e., no special target flags are needed to exploit the available performance. Supported GEMM datatypes are: FP64, FP32, FP16, bfloat16, BF8, HF8, MXBF8, MXHF8, MXINT8, int16, int8, MXBF6, MXHF6, MXFP4, int4, int2 and int1. Additionally, various non-standard low precision combinations are supported.

LIBXSMM 2.0: This major release repositions LIBXSMM as the reference implementation of Tensor Processing Primitives. To keep the core library focused, several application-specific pieces that previously shipped here have moved to dedicated companion repositories: the deep-learning operators and convolution drivers (LIBXSMM-DNN), the PyTorch integration (TPP PyTorch Extension), and the spectral-element reproducers (e.g. NekBox). The core library, the JIT backend, and the TPP frontend remain here.

For a list questions and answers, please also have a look at https://github.com/libxsmm/libxsmm/wiki/Q&A.

Getting Started: The following C code may be considered as Hello LIBXSMM. It walks through the Tensor Processing Primitives: dispatch a GEMM kernel and three element-wise (unary, binary, ternary) kernels, then invoke them. Build the example with cd /path/to/libxsmm; make STATIC=0 (shared library), save the code under hello.c (below) and compile with gcc -I/path/to/libxsmm/include hello.c -L/path/to/libxsmm/lib -lxsmm -lm -o hello, and finally execute with LD_LIBRARY_PATH=/path/to/libxsmm/lib LIBXSMM_VERBOSE=2 ./hello.

…

C /** Initialize the library; pay for setup cost at a specific point. / void libxsmm_init(void); /* De-initialize the library and free internal memory (optional). */ void libxsmm_finalize(void);

…

bash make STATIC=0 make


On CRAY systems, the CRAY Compiling Environment (CCE) should be used regardless of utilizing the CRAY compiler, the Intel Compiler, or the GNU&#160;Compiler Collection (GCC). The CCE is eventually suppressing to build shared libraries (STATIC=0). In any case, (1)&#160;switch to the desired compiler (module load/switch), and (2)&#160;rely on:

```bash
make CXX=CC CC=cc FC=ftn

A variety of build environments is out-of-the-box compatible, see https://github.com/libxsmm/libxsmm/wiki/Compatibility. If the build process is not successful, it may help to avoid advanced GCC flags. This is useful with a tool chain, which pretends to be GCC-compatible (and is treated as such) but fails to consume the afore mentioned flags:

bash

make COMPATIBLE=1

…

bash
make INTRINSICS=1

To test and validate a build, please consult https://github.com/libxsmm/libxsmm/wiki/Validation. To run some basic sanity checks, remember that each set of given key-value pairs represents a different build (and test):

bash

make STATIC=0 tests

To remove intermediate files, or to remove all generated files and folders (including the interface and the library archives), run one of the make-targets below. An additional distclean-target recursively cleans the entire tree (after version 1.9).

bash

make clean
make realclean

…

bash
cmake -S . -B build -DCMAKE_INSTALL_PREFIX=/path/to/libxsmm-install
cmake --build build --target install -j

The default CMake build produces static libraries. Add -DBUILD_SHARED_LIBS=ON to build shared libraries. When a Fortran compiler is detected, the Fortran module interface and libxsmmf are built automatically.

The installed package should be consumed through its exported targets:

cmake

find_package(libxsmm CONFIG REQUIRED)
target_link_libraries(my_target PRIVATE libxsmm::libxsmm)

For the Fortran module interface:

cmake

find_package(libxsmm CONFIG REQUIRED COMPONENTS Fortran)
target_link_libraries(my_target PRIVATE libxsmm::libxsmmf)

…

bash

bash
export PKG_CONFIG_PATH=/path/to/libxsmm/lib
pkg-config libxsmm --libs

Installation

There are two main mechanisms to install LIBXSMM (both mechanisms can be combined): (1) building the library in an out‑of‑tree fashion, and (2) installing into a certain location. Building in an out‑of‑tree fashion looks like:

bash

cd libxsmm-install
make -f /path/to/libxsmm/Makefile

Installation into a specific location looks like (PREFIX or DESTDIR):

bash

make MNK="1 2 3 4 5" PREFIX=/path/to/libxsmm-install install

…

bash
make PREFIX=/path/to/libxsmm-install STATIC=0 install
make PREFIX=/path/to/libxsmm-install install

…

bash
LIBXSMM_SIGNAL=0 ./myapplication

…

bash
LIBXSMM_VERBOSE=1 ./myapplication
[... application output]

HSW/SP      TRY    JIT    STA    COL
   0..13      0      0      0      0
  14..23      0      0      0      0
 24..128      3      3      0      0

…

bash
Registry: 20 MB (gemm=0 mcopy=14 tcopy=0)

If the call-wrapper is used, an additional runtime statistic becomes available (see Call Wrapper).

Note: Setting LIBXSMM_VERBOSE to a negative value dumps each generated JIT kernel to a file (binary) with each file being named like the function name shown in Intel VTune. Disassembly of the raw binary files can be accomplished by:

bash

objdump -D -b binary -m i386 -M x86-64 [JIT-dump-file]

Call Trace

During the initial steps of employing the LIBXSMM API, one may rely on a debug version of the library (make DBG=1). The latter also implies console output (stderr) in case of an error/warning condition inside of the library. It is also possible to print the execution flow (call trace) inside of LIBXSMM (can be combined with DBG=1 or OPT=0):

bash

make TRACE=1

Building an application which traces calls (inside of the library) requires the shared library of LIBXSMM, alternatively the application is required to link the static library of LIBXSMM in a dynamic fashion (GNU tool chain: -rdynamic). Tracing calls (without debugger) can be then accomplished by an environment variable called LIBXSMM_TRACE.

bash

LIBXSMM_TRACE=1 ./myapplication

Syntactically up to three arguments separated by commas (which allows to omit arguments) are taken (tid,i,n): tid signifies the ID of the thread to be traced with 1...NTHREADS being valid and where LIBXSMM_TRACE=1 is filtering for the "main thread" (in fact the first thread running into the trace facility); grabbing all threads (no filter) can be achieved by supplying a negative id (which is also the default when omitted). The second argument is pruning higher levels of the call-tree with i=1 being the default (level zero is the highest at the same level as the main function). The last argument is taking the number of inclusive call levels with n=-1 being the default (signifying no filter).

Although the ltrace (Linux utility) provides similar insight, the trace facility might be useful due to the afore mentioned filtering expressions. Please note that the trace facility is severely impacting the performance (even with LIBXSMM_TRACE=0), and this is not just because of console output but rather since inlining (internal) functions might be prevented along with additional call overhead on each function entry and exit. Therefore, debug symbols can be also enabled separately (make SYM=1; implied by TRACE=1 or DBG=1) which might be useful when profiling an application.

Verification

This section refers to testing correctness of an application using LIBXSMM utilities, i.e., using libxsmm_matdiff or libxsmm_matdiff_epsilon in particular. The former function (libxsmm_matdiff) compares two matrices (which can degenerate to vector shape), and yields a structure with information about the difference of both matrices (gold vs. test). The latter function (libxsmm_matdiff_epsilon) combines absolute and relative norms (given by afore mentioned structure) and calculates a scalar "epsilon" which can be used to check against a margin.

Using libxsmm_matdiff_epsilon in an application exposes an environment variable LIBXSMM_MATDIFF which can specify a file or directory path (LIBXSMM_MATDIFF=1 simply uses some filename as default). In any case, the application appends one line to the respective file for each call of libxsmm_matdiff_epsilon. A data record consists of the epsilon and the command line used to launch the application. A generated file can be further evaluated, e.g., sort -gk1 libxsmm_matdiff.log | tail -n 10 which yields the largest ten epsilon values discovered along with the application's command line.

The environment variable LIBXSMM_MATDIFF can carry optional space-separated arguments to amend each file entry like export LIBXSMM_MATDIFF="libxsmm_matdiff.log hello world". In sophisticated cases this can be used to amend a value only known at runtime, e.g., the actual margin which is used to judge the epsilon (putenv).

Performance

Profiling an application, which uses LIBXSMM's JIT-code is well-supported. The library supports Intel VTune Amplifier and Linux perf. Details are given on how to include profiler support, and how to run the application.

At build time, a variety of options exist to customize LIBXSMM. The library is setup for a broad range of use cases, which include sophisticated defaults for typical use.

Applications

High Performance Computing (HPC)

[1] https://cp2k.org/: Open Source Molecular Dynamics and the DBCSR library, which processes batches of small matrix multiplications. The batches originate from a distributed block-sparse matrix with problem-specific small matrices. Starting with CP2K 3.0, LIBXSMM can substitute CP2K's libsmm library.

[2] [https://gi