typical

Typical: data interchange with algebraic data types

Typical is a data serialization framework. You define data types in a file called a schema, then Typical generates efficient serialization and deserialization code for various languages. The generated code can be used for marshalling messages between services, storing structured data on disk, etc. The compact binary encoding supports forward and backward compatibility between different versions of your schema to accommodate evolving requirements.

Typical can be compared to Protocol Buffers and Apache Thrift. The main difference is that Typical has a more modern type system based on algebraic data types, emphasizing a safer programming style with non-nullable types and exhaustive pattern matching. You'll feel at home if you have experience with languages that embrace this style, such as Rust and Haskell. Typical proposes a new solution ("asymmetric" fields) to the classic problem of how to safely add or remove fields in record types without breaking compatibility. The concept of asymmetric fields also solves the dual problem of how to preserve compatibility when adding or removing cases in sum types.

In short, Typical offers two important features that are conventionally thought to be at odds:

1. Uncompromising type safety
2. Binary compatibility between schema versions

Typical's design was informed by experience using Protocol Buffers at Google and Apache Thrift at Airbnb. This is not an officially supported product of either company. If you want to support Typical, you can do so here.

Supported programming languages

The following languages are currently supported:

• Rust
• TypeScript
• JavaScript (via TypeScript)

Tutorial

To understand what this is all about, let's walk through an example scenario. Suppose you want to build a simple API for sending emails, and you need to decide how requests and responses will be serialized over the wire. You could use a self-describing format like JSON or XML, but you may want better type safety and performance. Typical has a great story to tell about those things.

Although our example scenario involves a client talking to a server, Typical has no notion of clients or servers. It only deals with serialization and deserialization. Other concerns like networking, encryption, and authentication are outside Typical's purview.

Step 1: Write a schema

You can start by creating a schema file called types.t (or any other name you prefer) with some types for your API:

struct SendEmailRequest {
    to: String = 0
    subject: String = 1
    body: String = 2
}

choice SendEmailResponse {
    success = 0
    error: String = 1
}

This schema defines two types: SendEmailRequest and SendEmailResponse. The first type is a struct, which means it describes messages containing a fixed set of fields (in this case, to, subject, and body). The second type is a choice, which means it describes messages containing exactly one field from a fixed set of possibilities (in this case, success and error). Structs and choices are called algebraic data types since they can be understood abstractly as multiplication and addition of types, respectively, but you don't need to know anything about that to use Typical.

Each field has both a name (e.g., body) and an integer index (e.g., 2). The name is just for humans, as only the index is used to identify fields in the binary encoding. You can freely rename fields without worrying about binary incompatibility, as long as you don't change the indices.

Each field also has a type (e.g., String). If the type is missing, as it is for the success field above, then it defaults to a built-in type called Unit. The Unit type holds no information and takes zero bytes to encode.

Once you've written your schema, Typical can format it to ensure consistent orthography such as indentation, letter case, etc. The following command will do it, though it won't have any effect on our example since it's already formatted properly:

typical format types.t

Step 2: Generate code for serialization and deserialization

Now that we've defined some types, we can use Typical to generate the code for serialization and deserialization. For example, you can generate Rust and TypeScript code with the following:

typical generate types.t --rust-file rust/types.rs --typescript-dir typescript

Refer to the example projects for how to automate this. In summary:

• For Rust, you can use a Cargo build script that is executed when you invoke cargo build.
• For TypeScript, you can use the scripts property of your package.json.

It's not necessary to set up an automated build system to use Typical, but we recommend doing so for convenience.

Step 3: Serialize and deserialize messages

With the code generated in the previous section, let's write a simple Rust program to serialize a message. We can write the message to an in-memory buffer, a socket, or anything that implements std::io::Write. For this example, we'll stream the data to a file.

let message = SendEmailRequestOut {
    to: "[email protected]".to_owned(),
    subject: "I love Typical!".to_owned(),
    body: "It makes serialization easy and safe.".to_owned(),
};

let mut file = BufWriter::new(File::create(REQUEST_FILE_PATH)?);
message.serialize(&mut file)?;
file.flush()?;

Another program could read the file and deserialize the message as follows:

let file = BufReader::new(File::open(FILE_PATH)?);
let message = SendEmailRequestIn::deserialize(file)?;

println!("to: {}", message.to);
println!("subject: {}", message.subject);
println!("body: {}", message.body);

The full code for this example can be found here. The TypeScript version is here.

We'll see in the next section why our SendEmailRequest type turned into SendEmailRequestOut and SendEmailRequestIn.

Required, optional, and asymmetric fields

Fields are required by default. This is an unusual design decision, since required fields are often thought to cause trouble for backward and forward compatibility between schema versions. Let's explore this topic in detail and see how Typical deals with it.

Adding or removing required fields is risky

Experience has taught us that it can be difficult to introduce a required field to a type that is already being used. For example, suppose your email API is up and running, and you want to add a new from field to the request type:

struct SendEmailRequest {
    to: String = 0

    # A new required field
    from: String = 3

    subject: String = 1
    body: String = 2
}

The only safe way to roll out this change (as written) is to finish updating all clients before beginning to update any servers. Otherwise, a client still running the old code might send a request to an updated server, which promptly rejects the request because it lacks the new field.

That kind of rollout may not be feasible. You might not be in control of the order in which clients and servers are updated. Or, perhaps the clients and servers are updated together, but not atomically. The client and the server might even be part of the same replicated service, so it wouldn't be possible to update one before the other no matter how careful you are.

Removing a required field can present analogous difficulties. Suppose, despite the aforementioned challenges, you were able to successfully introduce from as a required field. Now, an unrelated issue is forcing you to roll it back. That's just as dangerous as adding it was in the first place: if a client gets updated before a server, that client may then send the server a message without the from field, which the server will reject since it still expects that field to be present.

Promoting optional fields to required or vice versa is risky

A somewhat safer way to introduce a required field is to first introduce it as optional, and later promote it to required. For example, you can safely introduce this change:

struct SendEmailRequest {
    to: String = 0

    # A new optional field
    optional from: String = 3

    subject: String = 1
    body: String = 2
}

You would then update clients to set the new field. Once you're confident that the new field is always being set, you can promote it to required.

The trouble is that, as long as the field is optional, you can't rely on the type system to ensure the new field is always being set. Even if you're confident you've updated the client code appropriately, a teammate unaware of your efforts might introduce a new instance of the field being unset before you have the chance to promote it to required.

You can run into similar trouble when demoting a required field to optional. Once the field has been demoted, clients might stop setting the field before the servers can handle its absence, unless you can be sure the servers are updated first.

Making every field optional isn't ergonomic or safe

Due to the trouble associated with required fields, the conventional wisdom is simply to never use them; all fields should be declared optional. For example:

struct SendEmailRequest {
    optional to: String = 0
    optional subject: String = 1
    optional body: String = 2
}

However, this advice ignores the reality that some things really are semantically required, even if they aren't required according to the schema. An API cannot be expected to work if it doesn't have the data it needs. Having semantically required fields declared optional places extra burden on both writers and readers: writers can't rely on the type system to prevent them from accidentally forgetting to set the fields, and readers must address the case of the fields being missing to satisfy the type checker even though those fields are always supposed to be set.

Asymmetric fields can safely be promoted to required and vice versa

To help you safely add and remove required fields, Typical offers an intermediate state between optional and required: asymmetric. An asymmetric field in a struct is considered required for the writer, but optional for the reader. Unlike optional fields, an asymmetric field can safely be promoted to required and vice versa.

Let's make that more concrete with our email API example. Instead of directly introducing the from field as required, we first introduce it as asymmetric:

struct SendEmailRequest {
    to: String = 0

    # A new asymmetric field
    asymmetric from: String = 3

    subject: String = 1
    body: String = 2
}

Let's take a look at the generated code for this schema; we'll use Rust for this example. The generated code has two flavors of our SendEmailRequest type, one for serialization and another for deserialization:

pub struct SendEmailRequestOut {
    pub to: String,
    pub from: String,
    pub subject: String,
    pub body: String,
}

pub struct SendEmailRequestIn {
    pub to: String,
    pub from: Option,
    pub subject: String,
    pub body: String,
}

impl Serialize for SendEmailRequestOut {
    // Serialization code omitted.
}

impl Deserialize for SendEmailRequestIn {
    // Deserialization code omitted.
}

We can see the effect of from being an asymmetr