ZeroFormatter

ZeroFormatter

Fastest C# Serializer and Infinitely Fast Deserializer for .NET, .NET Core and Unity.

Note: this is unfair comparison, please see the performance section for the details.

Why use ZeroFormatter?

• Fastest C# serializer, the code is extremely tuned by both implementation and binary layout(see: performance)
• Deserialize/re-serialize is Infinitely fast because formatter can access to serialized data without parsing/packing(see: architecture)
• Strongly Typed and C# Code as schema, no needs to other IDL like .proto, .fbs...
• Smart API, only to use Serialize<T> and Deserialize<T>
• Full set of general purpose, multifunctional serializer, supports Union(Polymorphism) and native support of Dictionary, MultiDictionary(ILookup)
• First-class support to Unity(IL2CPP), it's faster than native JsonUtility

ZeroFormatter is similar as FlatBuffers but ZeroFormatter has clean API(FlatBuffers API is too ugly, see: sample; we can not use regularly) and C# specialized. If you need to performance such as Game, Distributed Computing, Microservices, etc..., ZeroFormatter will help you.

Install

for .NET, .NET Core

• PM> Install-Package ZeroFormatter

for Unity(Interfaces can reference both .NET 3.5 and Unity for share types), Unity binary exists on ZeroFormatter/Releases as well. More details, please see the Unity-Supports section.

• PM> Install-Package ZeroFormatter.Interfaces
• PM> Install-Package ZeroFormatter.Unity

Visual Studio Analyzer

• PM> Install-Package ZeroFormatter.Analyzer

Quick Start

Define class and mark as [ZeroFormattable] and public properties mark [Index] and declare virtual, call ZeroFormatterSerializer.Serialize<T>/Deserialize<T>.

…

Serializable target must mark ZeroFormattableAttribute, there public property must be virtual and requires IndexAttribute.

Analyzer

ZeroFormatter.Analyzer helps object definition. Attributes, accessibility etc are detected and it becomes a compiler error.

If you want to allow a specific type (for example, when registering a custom type), put ZeroFormatterAnalyzer.json at the project root and make the Build Action to AdditionalFiles.

This is a sample of the contents of ZeroFormatterAnalyzer.json.

[ "System.Uri" ]

Built-in support types

All primitives, All enums, TimeSpan, DateTime, DateTimeOffset, Guid, Tuple<,...>, KeyValuePair<,>, KeyTuple<,...>, Array, List<>, HashSet<>, Dictionary<,>, ReadOnlyCollection<>, ReadOnlyDictionary<,>, IEnumerable<>, ICollection<>, IList<>, ISet<,>, IReadOnlyCollection<>, IReadOnlyList<>, IReadOnlyDictionary<,>, ILookup<,> and inherited ICollection<> with paramterless constructor. Support type can extend easily, see: Extensibility section.

Define object rules

There rules can detect ZeroFormatter.Analyzer.

• Type must be marked with ZeroformattableAttribute.
• Public property must be marked with IndexAttribute or IgnoreFormatAttribute.
• Public property's must needs both public/protected get and set accessor.
• Public property's accessor must be virtual.
• Class is only supported public property not field(If struct can define field).
• IndexAttribute is not allowed duplicate number.
• Class must needs a parameterless constructor.
• Struct index must be started with 0 and be sequential.
• Struct needs full parameter constructor of index property types.
• Union type requires UnionKey property.
• UnionKey does not support multiple keys.
• All Union sub types must be inherited type.

The definition of struct is somewhat different from class.

[ZeroFormattable]
public struct Vector2
{
    [Index(0)]
    public float x;
    [Index(1)]
    public float y;

    // arg0 = Index0, arg1 = Index1
    public Vector2(float x, float y)
    {
        this.x = x;
        this.y = y;
    }
}

Struct index must be started with 0 and be sequential and needs full parameter constructor of index property types.

eager/lazy-evaluation

ZeroFormatter has two types of evaluation, "eager-evaluation" and "lazy-evaluation". If the type is lazy-evaluation, deserialization will be infinitely fast because it does not parse. If the user-defined class or type is IList<>, IReadOnlyList<>, ILazyLookup<>, ILazyDicitonary<>, ILazyReadOnlyDictionary<>, deserialization of that type will be lazily evaluated.

// MyClass is lazy-evaluation, all properties are lazily
[ZeroFormattable]
public class MyClass
{
    // int[] is eager-evaluation, when accessing Prop2, all values are deserialized
    [Index(0)]
    public virtual int[] Prop1 { get; set; }

    // IList<int> is lazy-evaluation, when accessing Prop2 with indexer, only that index value is deserialized 
    [Index(1)]
    public virtual IList<int> Prop2 { get; set; }
}

If you want to maximize the power of lazy-evaluation, define all collections with IList<>/IReadOnlyList<>.

ILazyLookup<>, ILazyDicitonary<>, ILazyReadOnlyDictionary<> is special collection interface, it defined by ZeroFormatter. The values defined in these cases are deserialized very quickly because the internal structure is also serialized in its entirety and does not need to be rebuilt data structure. But there are some limitations instead. Key type must be primitive, enum or there KeyTuple only because the key must be deterministic.

[ZeroFormattable]
public class MyClass
{
    [Index(0)]
    public virtual ILazyDictionary<int, int> LazyDictionary { get; set; }

    [Index(1)]
    public virtual ILazyLookup<int, int> LazyLookup { get; set; }
}

// there properties can set from `AsLazy***` extension methods. 

var mc = new MyClass();

mc.LazyDictionary = Enumerable.Range(1, 10).ToDictionary(x => x).AsLazyDictionary();
mc.LazyLookup = Enumerable.Range(1, 10).ToLookup(x => x).AsLazyLookup();

As a precaution, the binary size will be larger because all internal structures are serialized. This is a tradeoff, please select the best case depending on the situation.

Architecture

When deserializing an object, it returns a byte[] wrapper object. When accessing the property, it reads the data from the offset information of the header(and cache when needed).

Why must we define object in virtual? The reason is to converts access to properties into access to byte buffers.

If there is no change in data, reserialization is very fast because it writes the internal buffer data as it is. All serialized data can mutate and if the property type is fixed-length(primitive and some struct), it is written directly to internal binary data so keep the reserialization speed. If property is variable-length(string, list, object, etc...) the type and property are marked dirty. And it serializes only the difference, it is faster than normal serialization.

If property includes array/collection, ZeroFormatter can not track data was mutated so always marks dirty initially even if you have not mutated it. To avoid it, declare all collections with IList<> or IReadOnlyList<>.

If you want to define Immutable, you can use "protected set" and "IReadOnlyList<>".

[ZeroFormattable]
public class ImmutableClass
{
    [Index(0)]
    public virtual int ImmutableValue { get; protected set; }

    // IReadOnlyDictionary, ILazyReadOnlyDictionary, etc, too.
    [Index(1)]
    public virtual IReadOnlyList<int> ImmutableList { get; protected set; }
}

Binary size is slightly larger than Protobuf, MsgPack because of needs the header index area and all primitives are fixed-length(same size as FlatBuffers, smaller than JSON). It is a good idea to compress it to shrink the data size, gzip or LZ4(recommended, LZ4 is fast compression/decompression algorithm).

Versioning

If schema is growing, you can add Index.

[ZeroFormattable]
public class Version1
{
    [Index(0)]
    public virtual int Prop1 { get; set; }
    [Index(1)]
    public virtual int Prop2 { get; set; }
    
    // If deserialize from new data, ignored.
}

[ZeroFormattable]
public class Version2
{
    [Index(0)]
    public virtual int Prop1 { get; set; }
    [Index(1)]
    public virtual int Prop2 { get; set; }
    // You can add new property. If deserialize from old data, value is assigned default(T).
    [Index(2)]
    public virtual int NewType { get; set; }
}

But you can not delete index. If that index is unnecessary, please make it blank(such as [0, 1, 3]).

Only class definition is supported for versioning. Please note that struct is not supported.

DateTime

DateTime is serialized to UniversalTime so lose the TimeKind. If you want to change local time, use ToLocalTime after converted.

// in Tokyo, Japan Local Time(UTC+9)
var date = new DateTime(2000, 1, 1, 0, 0, 0, DateTimeKind.Local);
Console.WriteLine(date);

// 1999/12/31 15:00:00(UTC)
var deserialized = ZeroFormatterSerializer.Deserialize<DateTime>(ZeroFormatterSerializer.Serialize(date));

// 2000/1/1 00:00:00(in Tokyo, +9:00)
var toLocal = deserialized.ToLocalTime();

If you want to save offset info, use DateTimeOffset instead of DateTime.

// in Tokyo, Japan Local Time(UTC+9)
var date = new DateTime(2000, 1, 1, 0, 0, 0, DateTimeKind.Local);

// 2000/1/1, +9:00
var target = new DateTimeOffset(date);

// 2000/1/1, +9:00
var deserialized = ZeroFormatterSerializer.Deserialize<DateTimeOffset>(ZeroFormatterSerializer.Serialize(target));

Union

ZeroFormatter supports Union(Polymorphic) type. It can define abstract class and UnionAttributes, UnionKeyAttribute.

…

You can use Union as following.

…

If an unknown identification key arrives, an exception is thrown by default. However, it is also possible to return the default type - fallbackType.

…

Union can construct on execution time. You can mark DynamicUnion and make resolver on AppendDynamicUnionResolver.

…

Unity Supports

Put the ZeroFormatter.dll and ZeroFormatter.Interfaces.dll, modify Edit -> Project Settings -> Player -> Optimization -> Api Compatibillity Level to .NET 2.0 or higher.

ZeroFormatter.Unity works on all platforms(PC, Android, iOS, etc...). But it can 'not' use dynamic serializer generation due to IL2CPP issue. But pre code generate helps it. Code Generator is located in packages\ZeroFormatter.Interfaces.*.*.*\tools\zfc.exe. zfc is using Roslyn so analyze source code, pass the target csproj.

…

Note: Some options is important for reduce code generation size and startup speed on IL2CPP, especially -f is recommend if you use only DefaultResolver.

// Simple Case:
zfc.exe -i "..\src\Sandbox.Shared.csproj" -o "ZeroFormatterGenerated.cs"

// with t, c
zfc.exe -i "..\src\Sandbox.Shared.csproj" -o "..\\unity\ZfcCompiled\ZeroFormatterGenerated.cs" -t "System.Uri" -c "UNITY"

// -s
zfc.exe -i "..\src\Sandbox.Shared.csproj" -s -o "..\\unity\ZfcCompiled\" 

zfc.exe can setup on csproj's PreBuildEvent(useful to generate file path under self project) or PostBuildEvent(useful to generate file path is another project).

Note: zfc.exe is currently only run on Windows. It is .NET Core's Roslyn workspace API limitation but I want to implements to all platforms...

Generated formatters must nee