fastjson

fastjson - fast JSON parser and validator for Go

Features

• Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks.
• Parses arbitrary JSON without schema, reflection, struct magic and code generation contrary to easyjson.
• Provides simple API.
• Outperforms jsonparser and gjson when accessing multiple unrelated fields, since fastjson parses the input JSON only once.
• Validates the parsed JSON unlike jsonparser and gjson.
• May quickly extract a part of the original JSON with Value.Get(...).MarshalTo and modify it with Del and Set functions.
• May parse array containing values with distinct types (aka non-homogenous types). For instance, fastjson easily parses the following JSON array [123, "foo", [456], {"k": "v"}, null].
• fastjson preserves the original order of object items when calling Object.Visit.

Known limitations

• Requies extra care to work with - references to certain objects recursively returned by Parser must be released before the next call to Parse. Otherwise the program may work improperly. The same applies to objects returned by Arena. Adhere recommendations from docs.
• Cannot parse JSON from io.Reader. There is Scanner for parsing stream of JSON values from a string.

Usage

One-liner accessing a single field:

	s := []byte(`{"foo": [123, "bar"]}`)
	fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))

	// Output:
	// foo.0=123

Accessing multiple fields with error handling:

…

See also examples.

Security

• fastjson shouldn't crash or panic when parsing input strings specially crafted by an attacker. It must return error on invalid input JSON.
• fastjson requires up to sizeof(Value) * len(inputJSON) bytes of memory for parsing inputJSON string. Limit the maximum size of the inputJSON before parsing it in order to limit the maximum memory usage.

Performance optimization tips

• Re-use Parser and Scanner for parsing many JSONs. This reduces memory allocations overhead. ParserPool may be useful in this case.
• Prefer calling Value.Get* on the value returned from Parser instead of calling Get* one-liners when multiple fields must be obtained from JSON, since each Get* one-liner re-parses the input JSON again.
• Prefer calling once Value.Get for common prefix paths and then calling Value.Get* on the returned value for distinct suffix paths.
• Prefer iterating over array returned from Value.GetArray with a range loop instead of calling Value.Get* for each array item.

Fuzzing

Install go-fuzz & optionally the go-fuzz-corpus.

go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build

Build using go-fuzz-build and run go-fuzz with an optional corpus.

mkdir -p workdir/corpus
cp $GOPATH/src/github.com/dvyukov/go-fuzz-corpus/json/corpus/* workdir/corpus
go-fuzz-build github.com/valyala/fastjson
go-fuzz -bin=fastjson-fuzz.zip -workdir=workdir

Benchmarks

Go 1.12 has been used for benchmarking.

Legend:

• small - parse small.json (190 bytes).

• medium - parse medium.json (2.3KB).

• large - parse large.json (28KB).

• canada - parse canada.json (2.2MB).

• citm - parse citm_catalog.json (1.7MB).

• twitter - parse twitter.json (617KB).

• stdjson-map - parse into a map[string]interface{} using encoding/json.

• stdjson-struct - parse into a struct containing a subset of fields of the parsed JSON, using encoding/json.

• stdjson-empty-struct - parse into an empty struct using encoding/json. This is the fastest possible solution for encoding/json, may be used for json validation. See also benchmark results for json validation.

• fastjson - parse using fastjson without fields access.

• fastjson-get - parse using fastjson with fields access similar to stdjson-struct.

…

Benchmark results for json validation:

…

FAQ

• Q: There are a ton of other high-perf packages for JSON parsing in Go. Why creating yet another package? A: Because other packages require either rigid JSON schema via struct magic and code generation or perform poorly when multiple unrelated fields must be obtained from the parsed JSON. Additionally, fastjson provides nicer API.

• Q: What is the main purpose for fastjson? A: High-perf JSON parsing for RTB and other JSON-RPC services.

• Q: Why fastjson doesn't provide fast marshaling (serialization)? A: Actually it provides some sort of marshaling - see Value.MarshalTo. But I'd recommend using quicktemplate for high-performance JSON marshaling :)

• Q: fastjson crashes my program! A: There is high probability of improper use.

  • Make sure you don't hold references to objects recursively returned by Parser / Scanner beyond the next Parser.Parse / Scanner.Next call if such restriction is mentioned in docs.
  • Make sure you don't access fastjson objects from concurrently running goroutines if such restriction is mentioned in docs.
  • Build and run your program with -race flag. Make sure the race detector detects zero races.
  • If your program continue crashing after fixing issues mentioned above, file a bug.