cbor-x

cbor-x

The cbor-x package is an extremely fast and conformant CBOR NodeJS/JavaScript implementation. Currently, it is over 3-10x faster than any other CBOR JS implementation (including cbor-js and cborg) and faster than most MessagePack encoders, Avro, and generally faster than native V8 JSON.stringify/parse, on NodeJS. It implements the CBOR format as specificed in RFC-8949, RFC-8746, RFC-8742, Packed CBOR, numerous registered IANA tag extensions (the x in cbor-x), and proposed optional record extension, for defining record structures that makes CBOR even faster and more compact, often over twice as fast as even native JSON functions, and 15-50% more compact. See the performance section for more details. Structured cloning (with support for cyclical references) is supported through these tag extensions.

Basic Usage

Install on NodeJS with:

npm i cbor-x

And import or require it for basic standard serialization/encoding (encode) and deserialization/decoding (decode) functions:

import { decode, encode } from 'cbor-x';
let serializedAsBuffer = encode(value);
let data = decode(serializedAsBuffer);

This encode function will generate standard CBOR without any extensions that should be compatible with any standard CBOR parser/decoder. It will serialize JavaScript objects as CBOR maps by default. The decode function will deserialize CBOR maps as an Object with the properties from the map. The cbor-x package runs on any modern JS platform, but does have additional optimizations for NodeJS usage (and will use a node addon for performance boost as an optional dependency).

Deno Usage

Cbor-x modules are standard ESM modules and can be loaded directly from the deno.land registry for cbor for use in Deno. The standard encode and decode functionality is available on Deno, like other platforms.

Streams

We can use the including streaming functionality (which further improves performance). The EncoderStream is a NodeJS transform stream that can be used to serialize objects to a binary stream (writing to network/socket, IPC, etc.), and the DecoderStream can be used to deserialize objects from a binary stream (reading from network/socket, etc.):

import { EncoderStream } from 'cbor-x';
let stream = new EncoderStream();
stream.write(myData);

Or for a full example of sending and receiving data on a stream:

import { EncoderStream } from 'cbor-x';
let sendingStream = new EncoderStream();
let receivingStream = new DecoderStream();
// we are just piping to our own stream, but normally you would send and
// receive over some type of inter-process or network connection.
sendingStream.pipe(receivingStream);
sendingStream.write(myData);
receivingStream.on('data', (data) => {
	// received data
});

The EncoderStream and DecoderStream instances will have also the record structure extension enabled by default (see below).

Iterables

In addition to using CBOR with streams, CBOR can also encode to an iterable that can be iterated as a sequence of binary chunks with encodeAsIterable, which facilitates progressive encoding:

import { encodeAsIterable } from 'cbor-x';

for (let binaryChunk of encodeAsIterable(data)){
	// progressively get binary chunks as data is encoded
}

And encodeAsAsyncIterable is also available, which returns an async iterable, and can be used to encode data from async iterables as well as Blob data.

import { encodeAsAsyncIterable } from 'cbor-x';

let data = { blob: new Blob(...) };
for await (let binaryChunk of encodeAsAsyncIterable(data)){
	// progressively get binary chunks as asynchronous data source is encoded
}

Deno Usage

Cbor-x modules are standard ESM modules and can be loaded directly from the deno.land registry for cbor for use in Deno. The standard pack/encode and unpack/decode functionality is available on Deno, like other platforms.

Browser Usage

Cbor-x works as standalone JavaScript as well, and runs on modern browsers. It includes a bundled script, at dist/index.js for ease of direct loading:

This is UMD based, and will register as a module if possible, or create a CBOR global with all the exported functions.

For module-based development, it is recommended that you directly import the module of interest, to minimize dependencies that get pulled into your application:

import { decode } from 'cbor-x/decode' // if you only need to decode

Structured Cloning

You can also use cbor-x for structured cloning. By enabling the structuredClone option, you can include references to other objects or cyclic references, and object identity will be preserved.For example:

let obj = {
};
obj.self = obj;
let encoder = new Encoder({ structuredClone: true });
let serialized = encoder.encode(obj);
let copy = encoder.decode(serialized);
copy.self === copy // true

This option is disabled by default because reference checking degrades performance (by about 25-30%). (Note this implementation doesn't serialize every class/type specified in the HTML specification since not all of them make sense for storing across platforms.)

cbor-x also preserves certain typed objects like Error, Set, RegExp and TypedArray instances, using registered CBOR tag extensions. This works with or without structured cloning enabled.

Record / Object Structures

There is a critical difference between maps (or dictionaries) that hold an arbitrary set of keys and values (JavaScript Map is designed for these), and records or object structures that have a well-defined set of fields. Typical JS objects/records may have many instances re(use) the same structure. By using the record extension, this distinction is preserved in CBOR and the encoding can reuse structures and not only provides better type preservation, but yield much more compact encodings and increase decoding performance by 2-3x. Cbor-x automatically generates record definitions that are reused and referenced by objects with the same structure. Records use CBOR's tags to align well CBOR's tag/extension mechanism. There are a number of ways to use this to our advantage. For large object structures with repeating nested objects with similar structures, simply serializing with the record extension can yield significant benefits. To use the record structures extension, we create a new Encoder instance. By default a new Encoder instance will have the record extension enabled:

import { Encoder } from 'cbor-x';
let encoder = new Encoder();
encoder.encode(myBigData);

Another way to further leverage the benefits of the cbor-x record structures is to use streams that naturally allow for data to reuse based on previous record structures. The stream classes have the record structure extension enabled by default and provide excellent out-of-the-box performance.

When creating a new Encoder, EncoderStream, or DecoderStream instance, we can enable or disable the record structure extension with the objectsAsMaps property. When this is true, the record structure extension will be disabled, and all objects will revert to being serialized using MessageMap maps, and all maps will be deserialized to JS Objects as properties (like the standalone encode and decode functions).

Streaming with record structures works by encoding a structure the first time it is seen in a stream and referencing the structure in later messages that are sent across that stream. When an encoder can expect a decoder to understand previous structure references, this can be configured using the sequential: true flag, which is auto-enabled by streams, but can also be used with Packr instances.

Shared Record Structures

Another useful way of using cbor-x, and the record extension, is for storing data in a databases, files, or other storage systems. If a number of objects with common data structures are being stored, a shared structure can be used to greatly improve data storage and deserialization efficiency. In the simplest form, provide a structures array, which is updated if any new object structure is encountered:

import { Encoder } from 'cbor-x';
let encoder = new Encoder({
	structures: [... structures that were last generated ...]
});

If you are working with persisted data, you will need to persist the structures data when it is updated. Cbor-x provides an API for loading and saving the structures on demand (which is robust and can be used in multiple-process situations where other processes may be updating this same structures array), we just need to provide a way to store the generated shared structure so it is available to deserialize stored data in the future:

import { Encoder } from 'cbor-x';
let encoder = new Encoder({
	getStructures() {
		// storing our data in file (but we could also store in a db or key-value store)
		return decode(readFileSync('my-shared-structures.cbor')) || [];
	},
	saveStructures(structures) {
		writeFileSync('my-shared-structures.cbor', encode(structures))
	},
	structures: []
});

Cbor-x will automatically add and saves structures as it encounters any new object structures (up to a limit of 64). It will always add structures in incremental/compatible way: Any object encoded with an earlier structure can be decoded with a later version (as long as it is persisted).

Reading Multiple Values

If you have a buffer with multiple values sequentially encoded, you can choose to parse and read multiple values. This can be done using the decodeMultiple function/method, which can return an array of all the values it can sequentially parse within the provided buffer. For example:

let data = new Uint8Array([1, 2, 3]) // encodings of values 1, 2, and 3
let values = decodeMultiple(data) // [1, 2, 3]

Alternately, you can provide a callback function that is called as the parsing occurs with each value, and can optionally terminate the parsing by returning false:

let data = new Uint8Array([1, 2, 3]) // encodings of values 1, 2, and 3
decodeMultiple(data, (value) => {
	// called for each value
	// return false if you wish to end the parsing
})

KeyMaps for Senml

KeyMaps can be used to remap properties of source Objects and Maps to numerical equivalents for more efficient encoding. The principle driver for this feature is to support application/senml+cborcontent-encoding as defined in https://datatracker.ietf.org/doc/html/rfc8428#section-6 for use in LWM2M application (see http://www.openmobilealliance.org/release/LightweightM2M/V1_2-20201110-A/HTML-Version/OMA-TS-LightweightM2M_Core-V1_2-20201110-A.html#7-4-7-0-747-SenML-CBOR)

Records are also supported in conjunction with keyMaps, but these are disabled by default when keyMaps are specified as use of the two features does not introduce any additional compression efficiency unless that the data arrays are quite large (> 10 items).

…

CBOR Packing

Packed CBOR is additional specification for CBOR which allows for compact encoding of data that has repeated values. Cbor-x supports decoding packed CBOR, no flags or options needed. Cbor-x can also optionally generate packed CBOR (with the pack option), which will cause the encoder to look for repeated strings in a data structure that is being encoded, and store the strings in a packed table that c