非常快速的 Python JSON 分析库
Fast JSON parsing library for Python, 7-12 times faster than standard Python JSON parser.
It is Python bindings for the simdjson using Cython.
Standard Python JSON parser (json.load() etc.) is relatively slow,
and if you need to parse large JSON files or a large number of small JSON files,
it may represent a significant bottleneck.
Whilst there are other fast Python JSON parsers, such as pysimdjson, libpy_simdjson or orjson, they don't reach the raw speed that is provided by the brilliant SIMDJSON project. SIMDJSON is C++ JSON parser based on SIMD instructions, reportedly the fastest JSON parser on the planet.
import cysimdjson
json_bytes = b'''
{
"foo": [1,2,[3]]
}
'''
parser = cysimdjson.JSONParser()
json_element = parser.parse(json_bytes)
# Access using JSON Pointer
print(json_element.at_pointer("/foo/2/0"))Note: parser object can be reused for maximum performance.
parser = cysimdjson.JSONParser()
json_parsed = parser.loads(json_bytes)
# Access in a Python way
print(json_parsed.json_parsed['foo'])The json_parsed is a read-only dictionary-like object, that provides an access to JSON data.
WARNING: This method of the access will be deprecated in the future, likely in favour of JSON Pointer.
The speed of cysimdjson is based on these assumptions:
JSONParser object is still alive (not destroyed), otherwise you will get ugly errorsIf your design is not aligned with these assumptions, cysimdjson is not a good choice.
JSONParser.parse(json_bytes)
Parse JSON json_bytes, represented as bytes.
JSONParser.parse_in_place(bytes)
Parse JSON json_bytes, represented as bytes, assuming that there is a padding expected by SIMDJSON.
This is the fastest parsing variant.
JSONParser.parse_string(string)
Parse JSON json_bytes, represented as str (string).
JSONParser.load(path)
pip3 install cysimdjsonProject cysimdjson is distributed via PyPI: https://pypi.org/project/cysimdjson/ .
If you want to install cysimdjson from source, you need to install Cython first: pip3 install cython.
…………----------------------------------------------------------------
# 'jsonexamples/verysmall.json' 7 bytes
----------------------------------------------------------------
* cysimdjson parse 3972376.53 EPS ( 1.00) 27.81 MB/s
* orjson loads 3637369.63 EPS ( 1.09) 25.46 MB/s
* libpy_simdjson loads 1774211.19 EPS ( 2.24) 12.42 MB/s
* pysimdjson parse 977530.90 EPS ( 4.06) 6.84 MB/s
* python json loads 527932.65 EPS ( 7.52) 3.70 MB/s
----------------------------------------------------------------
SIMDJSON: 3799392.10 EPSCPU: AMD EPYC 7452
More performance testing:
pip3 install orjson
pip3 install pysimdjson
pip3 install libpy_simdjson
python3 setup.py build_ext --inplace
PYTHONPATH=. python3 ./perftest/test_benchmark.pypython3 setup.py build_ext --inplace暂无开放 Issues,或尚未同步最近议题。