sonic-rs

sonic-rs

English | 中文

A fast Rust JSON library based on SIMD. It has some references to other open-source libraries like sonic_cpp, serde_json, sonic, simdjson, rust-std and more.

For Golang users to use sonic_rs, please see for_Golang_user.md

For users to migrate from serde_json to sonic_rs, can see serdejson_compatibility

Requirements/Notes

1. Faster in x86_64 or aarch64, other architecture is fallback and maybe very slower.

2. Requires Rust nightly version Support Stable Rust now.

3. Please add the compile options -C target-cpu=native

4. Should enable sanitize feature to avoid false-positive if you are using LLVM-sanitizer in your program. Don't enable this feature in production, since it will cause 30% performance loss in serialize.

Quick to use sonic-rs

To ensure that SIMD instruction is used in sonic-rs, you need to add rustflags -C target-cpu=native and compile on the host machine. For example, Rust flags can be configured in Cargo config.

Add sonic-rs in Cargo.toml

[dependencies]
sonic-rs = "0.3"

Features

1. Serde into Rust struct as serde_json and serde.

2. Parse/Serialize JSON for untyped sonic_rs::Value, which can be mutable.

3. Get specific fields from a JSON with the blazing performance.

4. Use JSON as a lazy array or object iterator with the blazing performance.

5. Support LazyValue, Number and RawNumber(just like Golang's JsonNumber) in default.

6. The floating parsing precision is as Rust std in default.

Benchmark

The main optimization in sonic-rs is the use of SIMD. However, we do not use the two-stage SIMD algorithms from simd-json. We primarily use SIMD in the following scenarios:

1. parsing/serialize long JSON strings
2. parsing the fraction of float number
3. Getting a specific elem or field from JSON
4. Skipping white spaces when parsing JSON

More details about optimization can be found in performance.md.

Benchmarks environment:

Architecture:        x86_64
Model name:          Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz

AArch64 benchmark data can be found in benchmark_aarch64.md.

Benchmarks:

• Deserialize Struct: Deserialize the JSON into Rust struct. The defined struct and testdata is from json-benchmark

• Deseirlize Untyped: Deseialize the JSON into an untyped document

The serialize benchmarks work oppositely.

All deserialized benchmarks enabled UTF-8 validation and enabled float_roundtrip in serde-json to get sufficient precision as Rust std.

Deserialize Struct

The benchmark will parse JSON into a Rust struct, and there are no unknown fields in JSON text. All fields are parsed into struct fields in the JSON.

Sonic-rs is faster than simd-json because simd-json (Rust) first parses the JSON into a tape, then parses the tape into a Rust struct. Sonic-rs directly parses the JSON into a Rust struct, and there are no temporary data structures. The flamegraph is profiled in the citm_catalog case.

cargo bench --bench deserialize_struct -- --quiet

…

Deserialize Untyped

The benchmark will parse JSON into a document. Sonic-rs seems faster for several reasons:

• There are also no temporary data structures in sonic-rs, as detailed above.
• Sonic-rs uses a memory arena for the whole document, resulting in fewer memory allocations, better cache-friendliness, and mutability.
• The JSON object in sonic_rs::Value is an array. Sonic-rs does not build a hashmap.

cargo bench --bench deserialize_value -- --quiet

…

Serialize Untyped

cargo bench --bench serialize_value -- --quiet

We serialize the document into a string. In the following benchmarks, sonic-rs appears faster for the twitter JSON. The twitter JSON contains many long JSON strings, which fit well with sonic-rs's SIMD optimization.

…

Serialize Struct

cargo bench --bench serialize_struct -- --quiet

The explanation is as mentioned above.

…

Get from JSON

cargo bench --bench get_from -- --quiet

The benchmark is getting a specific field from the twitter.json.

• sonic-rs::get_unchecked_from_str: without validate
• sonic-rs::get_from_str: with validate
• gjson::get_from_str: without validate

Sonic-rs utilize SIMD to quickly skip unnecessary fields in the unchecked case, thus enhancing the performance.

twitter/sonic-rs::get_unchecked_from_str
                        time:   [75.671 µs 76.766 µs 77.894 µs]
twitter/sonic-rs::get_from_str
                        time:   [430.45 µs 434.62 µs 439.43 µs]
twitter/gjson::get_from_str
                        time:   [359.61 µs 363.14 µs 367.19 µs]

Usage

Serde into Rust Type

Directly use the Deserialize or Serialize trait.

use sonic_rs::{Deserialize, Serialize}; 
// sonic-rs re-exported them from serde
// or use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct Person {
    name: String,
    age: u8,
    phones: Vec,
}

fn main() {
    let data = r#"{
  "name": "Xiaoming",
  "age": 18,
  "phones": [
    "+123456"
  ]
}"#;
    let p: Person = sonic_rs::from_str(data).unwrap();
    assert_eq!(p.age, 18);
    assert_eq!(p.name, "Xiaoming");
    let out = sonic_rs::to_string_pretty(&p).unwrap();
    assert_eq!(out, data);
}

Get a field from JSON

Get a specific field from a JSON with the pointer path. The return is a LazyValue, which is a wrapper of a raw valid JSON slice.

We provide the get and get_unchecked apis. get_unchecked apis should be used in valid JSON, otherwise it may return unexpected result.

…

Parse and Serialize into untyped Value

Parse a JSON into a sonic_rs::Value.

…

JSON Iterator

Parse an object or array JSON into a lazy iterator.

…

JSON LazyValue & Number & RawNumber

If we need to parse a JSON value as a raw string, we can use LazyValue.

If we need to parse a JSON number into an untyped type, we can use Number.

If we need to parse a JSON number without loss of precision, we can use RawNumber. It likes encoding/json.Number in Golang, and can also be parsed from a JSON string.

Detailed examples can be found in raw_value.rs and json_number.rs.

Error handle

Sonic's errors are followed as serde-json and have a display around the error position, examples in handle_error.rs.

FAQs

About UTF-8

By default, sonic-rs enable the UTF-8 validation, except for xx_unchecked APIs.

About floating point precision

By default, sonic-rs uses floating point precision consistent with the Rust standard library, and there is no need to add an extra float_roundtrip feature like serde-json to ensure floating point precision.

If you want to achieve lossless precision when parsing floating-point numbers, such as Golang encoding/json.Number and serde-json arbitrary_precision, you can use sonic_rs::RawNumber.

Acknowledgement

Thanks the following open-source libraries. sonic-rs has some references to other open-source libraries like sonic_cpp, serde_json, sonic, simdjson, yyjson, rust-std and so on.

We rewrote many SIMD algorithms from sonic-cpp/sonic/simdjson/yyjson for performance. We reused the de/ser codes and modified necessary parts from serde_json to make high compatibility with serde. We reused part codes about floating parsing from rust-std to make it more accurate.

Referenced papers:

1. Parsing Gigabytes of JSON per Second
2. JSONSki: streaming semi-structured data with bit-parallel fast-forwarding

Contributing

Please read CONTRIBUTING.md for information on contributing to sonic-rs.