Zero encoding extreme performance binary serializer for C# and Unity.
Zero encoding extreme performance binary serializer for C# and Unity.
Zero encoding extreme performance binary serializer for C# and Unity.
Compared with System.Text.Json, protobuf-net, MessagePack for C#, Orleans.Serialization. Measured by .NET 7 / Ryzen 9 5950X machine. These serializers have
IBufferWriter<byte>method, serialized usingArrayBufferWriter<byte>and reused to avoid measure buffer copy.
For standard objects, MemoryPack is x10 faster and x2 ~ x5 faster than other binary serializers. For struct array, MemoryPack is even more powerful, with speeds up to x50 ~ x200 greater than other serializers.
MemoryPack is my 4th serializer, previously I've created well known serializers, ZeroFormatter, Utf8Json, MessagePack for C#. The reason for MemoryPack's speed is due to its C#-specific, C#-optimized binary format and a well tuned implementation based on my past experience. It is also a completely new design utilizing .NET 7 and C# 11 and the Incremental Source Generator (.NET Standard 2.1 (.NET 5, 6) and there is also Unity support).
Other serializers perform many encoding operations such as VarInt encoding, tag, string, etc. MemoryPack format uses a zero-encoding design that copies as much C# memory as possible. Zero-encoding is similar to FlatBuffers, but it doesn't need a special type, MemoryPack's serialization target is POCO.
Other than performance, MemoryPack has these features.
IBufferWriter<byte>, ReadOnlySpan<byte>, ReadOnlySequence<byte>)This library is distributed via NuGet. For best performance, recommend to use .NET 7. Minimum requirement is .NET Standard 2.1.
PM> Install-Package MemoryPack
And also a code editor requires Roslyn 4.3.1 support, for example Visual Studio 2022 version 17.3, .NET SDK 6.0.401. For details, see the Roslyn Version Support document.
For Unity, the requirements and installation process are completely different. See the Unity section for details.
Define a struct or class to be serialized and annotate it with the [MemoryPackable] attribute and the partial keyword.
using MemoryPack;
[MemoryPackable]
public partial class Person
{
public int Age { get; set; }
public string Name { get; set; }
}
Serialization code is generated by the C# source generator feature which implements the IMemoryPackable<T> interface. In Visual Studio you can check a generated code by using a shortcut Ctrl+K, R on the class name and select *.MemoryPackFormatter.g.cs.
Call MemoryPackSerializer.Serialize<T>/Deserialize<T> to serialize/deserialize an object instance.
var v = new Person { Age = 40, Name = "John" };
var bin = MemoryPackSerializer.Serialize(v);
var val = MemoryPackSerializer.Deserialize<Person>(bin);
Serialize method supports a return type of byte[] as well as it can serialize to IBufferWriter<byte> or Stream. Deserialize method supports ReadOnlySpan<byte>, ReadOnlySequence<byte> and Stream. And there are alse non-generics versions.
These types can be serialized by default:
byte, int, bool, char, double, etc.)enum, Any user-defined struct which doesn't contain reference types)string, decimal, Half, Int128, UInt128, Guid, Rune, BigIntegerTimeSpan, DateTime, DateTimeOffset, TimeOnly, DateOnly, TimeZoneInfoComplex, Plane, Quaternion Matrix3x2, Matrix4x4, Vector2, Vector3, Vector4Uri, Version, StringBuilder, Type, BitArray, CultureInfoT[], T[,], T[,,], T[,,,], Memory<>, ReadOnlyMemory<>, ArraySegment<>, ReadOnlySequence<>Nullable<>, Lazy<>, KeyValuePair<,>, Tuple<,...>, ValueTuple<,...>List<>, LinkedList<>, Queue<>, Stack<>, HashSet<>, SortedSet<>, PriorityQueue<,>Dictionary<,>, SortedList<,>, SortedDictionary<,>, ReadOnlyDictionary<,> Collection<>, ReadOnlyCollection<>, ObservableCollection<>, ReadOnlyObservableCollection<>IEnumerable<>, ICollection<>, IList<>, IReadOnlyCollection<>, IReadOnlyList<>, ISet<>IDictionary<,>, IReadOnlyDictionary<,>, ILookup<,>, IGrouping<,>,ConcurrentBag<>, ConcurrentQueue<>, ConcurrentStack<>, ConcurrentDictionary<,>, BlockingCollection<>ImmutableList<>, etc.) and interfaces (IImmutableList<>, etc.)[MemoryPackable] class / struct / record / record struct[MemoryPackable] can annotate to any class, struct, record, record struct and interface. If a type is struct or record struct which contains no reference types (C# Unmanaged types) any additional annotation (ignore, include, constructor, callbacks) is not used, that serialize/deserialize directly from the memory.
Otherwise, by default, [MemoryPackable] serializes public instance properties or fields. You can use [MemoryPackIgnore] to remove serialization target, [MemoryPackInclude] promotes a private member to serialization target.
…
MemoryPack's code generator adds information about what members are serialized to the <remarks /> section. This can be viewed by hovering over the type with Intellisense.
All members must be memorypack-serializable, if not the code generator will emit an error.
MemoryPack has 35 diagnostics rules (MEMPACK001 to MEMPACK035) to be defined comfortably.
If target type is defined MemoryPack serialization externally and registered, use [MemoryPackAllowSerialize] to silent diagnostics.
[MemoryPackable]
public partial class Sample2
{
[MemoryPackAllowSerialize]
public NotSerializableType? NotSerializableProperty { get; set; }
}
Member order is important, MemoryPack does not serialize the member-name or other information, instead serializing fields in the order they are declared. If a type is inherited, serialization is performed in the order of parent → child. The order of members can not change for the deserialization. For the schema evolution, see the Version tolerant section.
The default order is sequential, but you can choose the explicit layout with [MemoryPackable(SerializeLayout.Explicit)] and [MemoryPackOrder()].
// serialize Prop0 -> Prop1
[MemoryPackable(SerializeLayout.Explicit)]
public partial class SampleExplicitOrder
{
[MemoryPackOrder(1)]
public int Prop1 { get; set; }
[MemoryPackOrder(0)]
public int Prop0 { get; set; }
}
MemoryPack supports both parameterized and parameterless constructors. The selection of the constructor follows these rules. (Applies to classes and structs).
[MemoryPackConstructor], use it.[MemoryPackConstructor] attribute must be applied to the desired constructor (the generator will not automatically choose one), otherwise the generator will emit an error.…
When serializing/deserializing, MemoryPack can invoke a before/after event using the [MemoryPackOnSerializing], [MemoryPackOnSerialized], [MemoryPackOnDeserializing], [MemoryPackOnDeserialized] attributes. It can annotate both static and instance (non-static) methods, and public and private methods.
…
Callbacks allows parameterless method and ref reader/writer, ref T value method. For example, ref callbacks can write/read custom header before serialization process.
…
If set a value to ref value, you can change the value used for serialization/deserialization. For example, instantiate from ServiceProvider.
…
By default, annotated [MemoryPackObject] type try to serialize its members. However, if a type is a collection (ICollection<>, ISet<>, IDictionary<,>), use GenerateType.Collection to serialize it correctly.
[MemoryPackable(GenerateType.Collection)]
public partial class MyList<T> : List<T>
{
}
[MemoryPackable(GenerateType.Collection)]
public partial class MyStringDictionary<TValue> : Dictionary<string, TValue>
{
}
MemoryPackable class can not define static constructor because generated partial class uses it. Instead, you can define a static partial void StaticConstructor() to do the same thing.
[MemoryPackable]
public partial class CctorSample
{
static partial void StaticConstructor()
{
}
}
MemoryPack supports serializing interface and abstract class objects for polymorphism serialization. In MemoryPack this feature is called Union. Only interfaces and abstracts classes are allowed to be annotated with [MemoryPackUnion] attributes. Unique union tags are required.
…
tag allows 0 ~ 65535, it is especially efficient for less than 250.
If an interface and derived types are in different assemblies, you can use MemoryPackUnionFormatterAttribute instead. Formatters are generated the way that they are automatically registered via ModuleInitializer in C# 9.0 and above.
Note that
ModuleInitializeris not supported in Unity, so the formatter must be manually registered. To register your union formatter invoke{name of your union formatter}Initializer.RegisterFormatter()manually in Startup. For exampleUnionSampleFormatterInitializer.RegisterFormatter().
// AssemblyA
[MemoryPackable(GenerateType.NoGenerate)]
public partial interface IUnionSample
{
}
// AssemblyB define definition outside of target type
[MemoryPackUnionFormatter(typeof(IUnionSample))]
[MemoryPackUnion(0, typeof(FooClass))]
[MemoryPackUnion(1, typeof(BarClass))]
public partial class UnionSampleFormatter
{
}
Union can be assembled in code via DynamicUnionFormatter<T>.
// (ushort, Type)[]
var formatter = new DynamicUnionFormatter<IFooBarBaz>(
(0, typeof(Foo)),
(1, typeof(Bar)),
(2, typeof(Baz))
);
MemoryPackFormatterProvider.Register(formatter);
Serialize has three overloads.
// Non generic API also available, these version is first argument is Type and value is object?
byte[] Serialize<T>(in T? value, MemoryPackSerializerOptions? options = default)
void Serialize<T, TBufferWriter>(in TBufferWriter bufferWriter, in T? value, MemoryPackSerializerOptions? options = default)
async ValueTask SerializeAsync<T>(Stream stream, T? value, MemoryPackSerializerOptions? options = default, CancellationToken cancellationToken = default)
For performance, the recommended API uses BufferWriter. This serializes directly into the buffer.
No open issues yet, or sync has not completed.