rapidyaml

Rapid YAML

Or ryml, for short. ryml is a C++ library to parse and emit YAML, and do it fast, on everything from x64 to bare-metal chips without operating system.

The library is fully conformant to the YAML 1.2 spec, and passes 100% of the cases in the YAML test suite. The parser is state-machine based and not recursive. No known vulnerabilities exist. The code is robust and extremely tested and fuzzed.

ryml parses both read-only and in-place source buffers; the resulting tree holds only views to sub-ranges of the source buffer. No string copies or duplications are done, and no virtual functions are used. Unlike parsing models that build heavy dictionary trees, ryml parses data into a flat, index-based data tree, making it highly efficient for massive datasets, and bare-metal systems.

With the tree, (de)serialization happens only at your direct request, after parsing / before emitting. Internally, the tree representation stores only string views and assumes nothing on user types. And it is not just parsing or emitting which is fast; the serialization is extremely fast, in many cases faster than the fastest C++ facilities like std::to_chars(). ryml makes it easy and fast to read and modify the data tree.

ryml can use custom global and per-tree memory allocators and error handler callbacks, and is exception-agnostic. ryml provides a default implementation for the allocator (using std::malloc()) and error handlers (using using either exceptions, longjmp() or std::abort()), but you can opt out and provide your own memory allocation and eg, exception-throwing callbacks.

ryml does not depend on the STL, ie, it does not use any std container as part of its data structures, but it can serialize and deserialize these containers into the data tree. ryml ships with parts of c4core, a small C++ utilities multiplatform library, but you can ignore the in-source version of c4core and use a custom or system-installed c4core version.

ryml is written in C++11, and compiles cleanly with:

• Visual Studio 2015 and later
• clang++ 3.9 and later
• g++ 4.8 and later
• Intel Compiler

ryml's CI pipeline is merciless and extremely demanding, covering Linux, Windows and MacOS. The tests include analysing ryml with valgrind, clang-tidy, and gcc/clang sanitizers (asan, ubsan, lsan), and cover most architectures: x64, x86, wasm (emscripten), aarch64, armv7, armv6, armv5, armv4, loongarch64, mips, mipsel, mips64, mips64el, riscv64, s390x, ppc, ppc64, ppc64le and sparc64. ryml also runs in bare-metal.

ryml is available in Python (secondary repo is rapidyaml-python), and can very easily be compiled to JavaScript through emscripten (see below).

See also the changelog and the roadmap.

License

ryml is permissively licensed under the MIT license.

YAML standard conformance

ryml is fully conformant to the YAML specification, and it passes all of the cases in the YAML test suite. There are some minor deliberate deviations in default behavior; refer to the relevant page for more information.

Is it rapid?

Talk is cheap, so here are some performance plots with an example yaml file using parse and emit benchmarks, comparing ryml with yamlcpp, libyaml and fyaml (click the image to see the data):

To prove that this is not coincidence for a particular file, here are scatter plots across YAML files with different styles and sizes. The ryml variants are ryml_yaml_inplace_reuse_reserve and ryml_yaml_str_reserve from the plots above (again, click the image to see the data):

In absolute terms, ryml parses at ~200MB/s and emits at ~600MB/s, and is generally 30x (parse) / 150x (emit) faster than yamlcpp, and never less than 10x (parse) / 50x (emit). These are stark numbers, and justify the library name. For more details, refer to this page.

JSON performance

ryml is also among the fastest JSON handlers. It may not be the fastest, but is well ahead of the pack; here are some performance results. The benchmark code is the same as above, and it is reading a compile_commands.json file. The json libraries are rapidjson, sajson, jsoncpp and nlohmann:

Parse performanceEmit performance

So ryml beats most json parsers at their own game, with the only exception of rapidjson. As for emitting, rapidyaml is hands-down the fastest, exceeding all JSON libraries. Even parsing full YAML is at ~200MB/s, which is still in the JSON performance ballpark, albeit at its lower end. This is something to be proud of, as the YAML specification is much more complex than JSON: 23449 vs 1969 words.

Serialization performance

Serialization speed also matters for overall speed. For (de)serialization ryml uses the charconv facilities from c4core; these functions are blazing fast, and generally outperform the fastest equivalent facilities in the standard library by a significant margin. Refer to its documentation. For example, here are some results for c4::xtoa():

gcc 12.1Visual Studio 2019

Companies and institutions using rapidyaml

• Google
• Nvidia: nsight-perf-sdk and Energon
• CERN: ROOT and LHCb
• opentelemetry.io (CNCF / Linux Foundation)
• Toyota
• Qrypt
• OneArc
• BrainGlobe

This list above was gathered in a quick web search. Please reach out if you'd rather not see yours listed here. And do reach out to let me know how ryml helped solve your problems; this motivates me to continue improving the library.

Success stories

• 200x faster read (parse+serialization)
• 67x faster parse (OPENXCOM)
• 10x to 70x faster parse (rATHENA)
• 35x faster serialization
• python:
  • 150x faster read (4C-multiphysics)
  • 25x faster emit

Testimonials

• I've switched from yaml-cpp to ryml and it went from an approx. speed of 0.5mb/s to 111mb/s, a total life saver. (...) it has been very helpful. (src)

• memory consumption (virtual) went from 1.1GB to 94MB, indicating less fragmentation. (src)

• Thanks again for your help and your amazing libraries! (...) here are my thoughts so far:

  • Better code verbosity
  • So much customisation !
  • Obviously speed!!!

  (src)

• awesome library for YAML parsing. I'm very happy with it. (src)

• thanks for the awesome library! (src)

• I am enjoying using the library to help my C++ applications safely load YAML configuration files (src)

• impressive library (src)

• Thanks for this impressive library! (src)

• rapidyaml combined with c4conf is extremely powerful (src)

Quick start

If you're wondering whether ryml's speed comes at a convenience cost, you need not. ryml was written with easy and efficient usage in mind:

// Parse YAML code in place, potentially mutating the buffer:
char yml_buf[] = "{foo: 1, bar: [2, 3], john: doe}";
ryml::Tree tree = ryml::parse_in_place(yml_buf);

// read from the tree:
ryml::NodeRef bar = tree["bar"];
CHECK(bar[0].val() == "2");
CHECK(bar[1].val() == "3");
CHECK(bar[0].val().str == yml_buf + 15); // points at the source buffer
CHECK(bar[1].val().str == yml_buf + 18);

// deserializing:
int bar0 = 0, bar1 = 0;
bar[0].load(&bar0); // also checks the node is readable, and conversion succeeded
bar[1].load(&bar1); // see also .deserialize()
CHECK(bar0 == 2);
CHECK(bar1 == 3);

// serializing:
bar[0].save(10); // creates a string in the tree's arena
bar[1].save(11); // see also .set_serialized()
CHECK(bar[0].val() == "10");
CHECK(bar[1].val() == "11");

// add nodes
tree["new"].set_val("node");
bar.append_child().save(12);
CHECK(bar[2].val() == "12");

// emit tree
std::string expected = "{foo: 1,bar: [10,11,12],john: doe,new: node}";
// emit tree to std::string
CHECK(ryml::emitrs_yaml(tree) == expected);
// emit tree to FILE*
ryml::emit_yaml(tree, stdout); printf("\n");
// emit tree to ostream
std::cout (foo) == expected);
// emit node to FILE*
ryml::emit_yaml(foo, stdout);
// emit node to ostream
std::cout  [!TIP]
> Do see [the quickstart sample](https://rapidyaml.readthedocs.io/latest/doxygen/group__doc__quickstart.html), which covers everything in the library. In particular, do not start writing code without first [reading the quickstart overview](https://rapidyaml.readthedocs.io/latest/doxygen/group__doc__quickstart__overview.html). 

## Using ryml in your project

There are [many ways you can use ryml in your project](https://rapidyaml.readthedocs.io/latest/sp