simdutf

simdutf: Text processing at billions of characters per second

• Real-World Usage
• How fast is it?
• Requirements
• Usage (Usage)
• Usage (CMake)
• Single-header version
• Single-header version with limited features
• Packages
• Example
• API
• Cost of the safe conversion functions
• Base64
• Find
• C++20 and std::span usage in simdutf
• C++23 and constexpr support
• Command-line tools
• Manual implementation selection
• Benchmarks
• Compiling without the C++ standard library
• C API
• SIMDUTF_USE_STATIC_INITIALIZATION
• Thread safety
• References
• License

Most modern software relies on the Unicode standard. In memory, Unicode strings are represented using either UTF-8 or UTF-16. The UTF-8 format is the de facto standard on the web (JSON, HTML, etc.) and it has been adopted as the default in many popular programming languages (Go, Zig, Rust, Swift, etc.). The UTF-16 format is standard in Java, C# and in many Windows technologies.

Not all sequences of bytes are valid Unicode strings. It is unsafe to use Unicode strings in UTF-8 and UTF-16LE without first validating them. Furthermore, we often need to convert strings from one encoding to another, by a process called transcoding. For security purposes, such transcoding should be validating: it should refuse to transcode incorrect strings.

This library provide fast Unicode functions such as

• ASCII, UTF-8, UTF-16LE/BE and UTF-32 validation, with and without error identification,
• Latin1 to UTF-8 transcoding,
• Latin1 to UTF-16LE/BE transcoding
• Latin1 to UTF-32 transcoding
• UTF-8 to Latin1 transcoding, with or without validation, with and without error identification,
• UTF-8 to UTF-16LE/BE transcoding, with or without validation, with and without error identification,
• UTF-8 to UTF-32 transcoding, with or without validation, with and without error identification,
• UTF-16LE/BE to Latin1 transcoding, with or without validation, with and without error identification,
• UTF-16LE/BE to UTF-8 transcoding, with or without validation, with and without error identification,
• UTF-32 to Latin1 transcoding, with or without validation, with and without error identification,
• UTF-32 to UTF-8 transcoding, with or without validation, with and without error identification,
• UTF-32 to UTF-16LE/BE transcoding, with or without validation, with and without error identification,
• UTF-16LE/BE to UTF-32 transcoding, with or without validation, with and without error identification,
• From an UTF-8 string, compute the size of the Latin1 equivalent string,
• From an UTF-8 string, compute the size of the UTF-16 equivalent string,
• From an UTF-8 string, compute the size of the UTF-32 equivalent string (equivalent to UTF-8 character counting),
• From an UTF-16LE/BE string, compute the size of the Latin1 equivalent string,
• From an UTF-16LE/BE string, compute the size of the UTF-8 equivalent string,
• From an UTF-32 string, compute the size of the UTF-8 or UTF-16LE equivalent string,
• From an UTF-16LE/BE string, compute the size of the UTF-32 equivalent string (equivalent to UTF-16 character counting),
• UTF-8 and UTF-16LE/BE character counting,
• UTF-16 endianness change (UTF16-LE/BE to UTF-16-BE/LE),
• WHATWG forgiving-base64 (with or without URL encoding) to binary,
• Binary to base64 (with or without URL encoding).

The functions are accelerated using SIMD instructions (e.g., ARM NEON, SSE, AVX, AVX-512, RISC-V Vector Extension, LoongSon, POWER, etc.). When your strings contain hundreds of characters, we can often transcode them at speeds exceeding a billion characters per second. You should expect high speeds not only with English strings (ASCII) but also Chinese, Japanese, Arabic, and so forth. We handle the full character range (including, for example, emojis).

The library compiles down to a small library of a few hundred kilobytes. Our functions are exception-free and non allocating. We have extensive tests and extensive benchmarks.

We have exhaustive tests, including an elaborate fuzzing setup. The library has been used in production systems for years.

If using C++23 or newer, there is experimental support for using the library at compile time (constexpr).

Real-World Usage

The simdutf library is used by:

• Node.js (19.4.0 or better, 20.0 or better, 18.15 or better), a standard JavaScript runtime environment,
• Bun, a fast JavaScript runtime environment,
• WebKit, the Web engine behind the Safari browser (iOS, macOS),
• Chromium, the Web engine behind the Google Chrome, Microsoft Edge and Brave,
• StarRocks, an Open-Source, High-Performance Analytical Database,
• Oracle GraalVM JavaScript, a JavaScript implementation by Oracle,
• Couchbase, a popular database system,
• Ladybird, an independent Web browser,
• Cloudflare workerd, a JavaScript/Wasm Runtime,
• haskell/text, a library for fast operations over Unicode text,
• klogg, a Really fast log explorer,
• Pixie, observability tool for Kubernetes applications,
• fluentbit, Fast and Lightweight Logs, Metrics and Traces processor for Linux, BSD, OSX and Windows,
• ghostty, Fast terminal emulator,
• uWebSockets, web server for the most demanding of applications,
• vte (0.81.0 or better), a virtual terminal widget for GTK applications.

How fast is it?

The adoption of the simdutf library by the popular Node.js JavaScript runtime lead to a significant performance gain:

Decoding and Encoding becomes considerably faster than in Node.js 18. With the addition of simdutf for UTF-8 parsing the observed benchmark, results improved by 364% (an extremely impressive leap) when decoding in comparison to Node.js 16. (State of Node.js Performance 2023)

Over a wide range of realistic data sources, the simdutf library transcodes a billion characters per second or more. Our approach can be 3 to 10 times faster than the popular ICU library on difficult (non-ASCII) strings. We can be 20x faster than ICU when processing easy strings (ASCII). Our good results apply to both recent x64 and ARM processors.

To illustrate, we present a benchmark result with values are in billions of characters processed by second. Consider the following figures.

If your system supports AVX-512, the simdutf library can provide very high performance. We get the following speed results on an Ice Lake Intel processor (both AVX2 and AVX-512) are simdutf kernels:

Datasets: https://github.com/lemire/unicode_lipsum

Please refer to our benchmarking tool for a proper interpretation of the numbers. Our results are reproducible.

Requirements

• C++17 compatible compiler. We support LLVM clang, GCC, Visual Studio. Be aware that GCC under Windows is buggy and thus unsupported.
• For high speed, you should have a recent 64-bit system (e.g., ARM, x64, RISC-V with vector extensions, Loongson, POWER). On Loongson processors, LASX runtime dispatching is only enabled on GCC 15+, not on LLVM or older versions of GCC.
• If you rely on CMake, you should use a recent CMake (at least 3.18); otherwise you may use the single header version. The library is also available from Microsoft's vcpkg, from conan, from FreeBSD's port, from brew, and many other systems.
• AVX-512 support require a processor with AVX512-VBMI2 (Ice Lake or better, AMD Zen 4 or better) and a recent compiler (GCC 8 or better, Visual Studio 2022 or better, LLVM clang 6 or better). You need a correspondingly recent assembler such as gas (2.30+) or nasm (2.14+): recent compilers usually come with recent assemblers. If you mix a recent compiler with an incompatible/old assembler (e.g., when using a recent compiler with an old Linux distribution), you may get errors at build time because the compiler produces instructions that the assembler does not recognize: you should update your assembler to match your compiler (e.g., upgrade binutils to version 2.30 or better under Linux) or use an older compiler matching the capabilities of your assembler.
• To benefit from RISC-V Vector Extensions on RISC-V systems, you should compile specifically for the desired architecture. E.g., add -march=rv64gcv as a compiler flag when using a version of GCC or LLVM which supports these extensions (such as GCC 14 or better). The command CXXFLAGS=-march=rv64gcv cmake -B build may suffice.
• We recommend that Visual Studio users compile with LLVM (ClangCL). Using LLVM as a front-end inside Visual Studio provides faster release builds and better runtime performance.

Usage (Usage)

We made a video to help you get started with the library.

Quick Start

Linux or macOS users might follow the following instructions if they have a recent C++ compiler installed and the standard utilities (wget, unzip, etc.)

1. Pull the library in a directory

   bash复制

   wget https://github.com/simdutf/simdutf/releases/download/v9.2.0/singleheader.zip
   unzip singleheader.zip

   You can replace wget by curl -OL https://... if you prefer.

2. Compile

   bash复制

   c++ -std=c++17 -o amalgamation_demo amalgamation_demo.cpp

3. ./amalgamation_demo

   复制

   valid UTF-8
   wrote 4 UTF-16LE words.
   valid UTF-16LE
   wrote 4 UTF-8 words.
   1234
   perfect round trip

We strongly discourage working from our main git branch. You should never use our main branch in production. Use our releases. They are tagged as vX.Y.Z.

Usage (CMake)

cmake -B build
cmake --build build
cd build
ctest .

Visual Studio users must specify whether they want to build the Release or Debug version.

To use the library as a CMake dependency in your project, please see tests/installation_tests/from_fetch for an example.

You may also use a package manager. E.g., we have a complete example using vcpkg.

Single-header version

You can create a single-header version of the library where all of the code is put into two files (simdutf.h and simdutf.cpp). We publish a zip archive containing these files, e.g., see https://github.com/simdutf/simdutf/releases/download/v9.2.0/singleheader.zip

You may generate it on your own using a Python script.

python3 ./singleheader/amalgamate.py

We require Python 3 or better.

Under Linux and m