hyperpb-go

hyperpb

hyperpb is a highly optimized dynamic message library for Protobuf or read-only workloads. It is designed to be a drop-in replacement for dynamicpb, protobuf-go's canonical solution for working with completely dynamic messages.

hyperpb's parser is an efficient VM for a special instruction set, a variant of table-driven parsing (TDP), pioneered by the UPB project.

Our parser is very fast, beating dynamicpb by 10x, and often beating protobuf-go's generated code by a factor of 2-3x, especially for workloads with many nested messages.

Here, we show two benchmark variants for hyperpb: out-of-the-box performance with no optimizations turned on, and real-time profile-guided optimization (PGO) with all optimizations we currently offer enabled. These benchmarks are a subset of the ones you can run using make bench.

You can read about hyperpb on the Buf blog, or on project lead Miguel's blog.

• Introducing hyperpb
• Parsing Protobuf Like Never Before
• Cheating the Reaper in Go

Usage

The core conceit of hyperpb is that you must pre-compile a parser using hyperpb.Compile at runtime, much like regular expressions require that you regexp.Compile them. Doing this allows hyperpb to run optimization passes on your message, and delaying it to runtime allows us to continuously improve layout optimizations, without making any source-breaking changes.

For example, let's say that we want to compile a parser for some type baked into our binary, and parse some data with it.

…

Currently, hyperpb only supports manipulating messages through the reflection API; it shines best when you need write a very generic service that downloads types off the network and parses messages using those types, which forces you to use reflection.

Mutation is currently not supported; any operation which would mutate an already-parsed message will panic. Which methods of *hyperpb.Message panic is included in the documentation.

Using types from a registry

We can use the hyperpb.CompileFileDescriptorSet function to parse a dynamic type and use it to walk the fields of a message:

…

Since any generic, non-mutating operation will work with hyperpb messages, we can use them as an efficient transcoding medium from the wire format, for runtime-loaded messages.

func dynamicMessageToJSON(
    schema *descriptorpb.FileDescriptorSet,
    messageName protoreflect.FullName,
    data []byte,
) ([]byte, error) {
    msgType, err := hyperpb.CompileFileDescriptorSet(schema, messageName)
    if err != nil {
        return nil, err
    }

    msg := hyperpb.NewMessage(msgType)
    if err := proto.Unmarshal(data, msg); err != nil {
        return nil, err
    }

    // Dump the message to JSON. This just works!
    return protojson.Marshal(msg)
}

protovalidate also works directly on reflection, so it works out-of-the-box:

func validateDynamicMessage(
    schema *descriptorpb.FileDescriptorSet,
    messageName protoreflect.FullName,
    data []byte,
) error {
    // Unmarshal like before.
    msgType, err := hyperpb.CompileFileDescriptorSet(schema, messageName)
    if err != nil {
        return err
    }

    msg := hyperpb.NewMessage(msgType)
    if err := proto.Unmarshal(data, msg); err != nil {
        return err
    }

    // Run custom validation. This just works!
    return protovalidate.Validate(msg)
}

Advanced Usage

hyperpb is all about parsing as fast as possible, so there are a number of optimization knobs available. Calling Message.Unmarshal directly instead of proto.Unmarshal allows setting custom UnmarshalOptions:

func unmarshalWithCustomOptions(
    schema *descriptorpb.FileDescriptorSet,
    messageName protoreflect.FullName,
    data []byte,
) error {
    msgType, err := hyperpb.CompileFileDescriptorSet(schema, messageName)
    if err != nil {
        return err
    }

    msg := hyperpb.NewMessage(msgType)
    return msg.Unmarshal(
        data,
        hyperpb.WithMaxDecodeMisses(16),
        // Additional options...
    )
}

The compiler also takes CompileOptions, such as for configuring how extensions are resolved:

msgType, err := hyperpb.CompileFileDescriptor(
    schema,
    messageName,
    hyperpb.WithExtensionsFromTypes(typeRegistry),
    // Additional options...
)

Memory Reuse

hyperpb also has a memory-reuse mechanism that side-steps the Go garbage collector for improved allocation latency. hyperpb.Shared is book-keeping state and resources shared by all messages resulting from the same parse. After the message goes out of scope, these resources are ordinarily reclaimed by the garbage collector.

However, a hyperpb.Shared can be retained after its associated message goes away, allowing for re-use. Consider the following example of a request handler:

type requestContext struct {
    shared *hyperpb.Shared
    types map[string]*hyperpb.MessageType
    // Additional context fields...
}

func (c *requestContext) Handle(req Request) {
    msgType := c.types[req.Type]
    msg := c.shared.NewMessage(msgType)
    defer c.shared.Free()

    c.process(msg, req, ...)
}

Beware that msg must not outlive the call to Shared.Free; failure to do so will result in memory errors that Go cannot protect you from.

Profile-Guided Optimization (PGO)

hyperpb supports online PGO for squeezing extra performance out of the parser by optimizing the parser with knowledge of what the average message actually looks like. For example, using PGO, the parser can predict the expected size of repeated fields and allocate more intelligently.

For example, suppose you have a corpus of messages for a particular type. You can build an optimized type, using that corpus as the profile, using Type.Recompile:

…

Using a custom sampling rate in hyperpb.WithRecordProfile, it's possible to sample data on-line as part of a request flow, and recompile dynamically:

…

Compatibility

hyperpb is experimental software, and the API may change drastically before v1. It currently implements all Protobuf language constructs. It does not implement mutation of parsed messages, however.

Supported Targets

hyperpb is currently only supported on 64-bit x86 and ARM targets (Go calls these amd64 and arm64). The library will not build on other architectures, and PRs to add new architectures without a way to run tests for them in CI will be rejected.

Support for 32-bit targets is unlikely to happen, even if we have test runners. hyperpb assumes a 64-bit general-purpose register width, and attempting to tune for a 32-bit register size will require a 32-bit re-implementation of the parser VM.

Similarly, we assume little-endian in many places for performance, particularly because Protobuf's wire format is little-endian. Getting big-endian support will be a lot of work and is unlikely to perform anywhere close to little-endian.

If you would like to try to use an architecture that we don't support, build with the hyperpb.unsupported tag. If it breaks, you get to keep both pieces: which is to say, issues stemming from use of this build tag will be closed.

Contributing

For a detailed explanation of the implementation details of hyperpb, see the DESIGN.md file. Contributions that significantly change the parser will require benchmarks; you can run them with make bench.

Legal

Offered under the Apache 2 license.