#1463·serde

Replacement API for Deserializer in-band Signalling

Author: mitsuhikoCreated Jan 25, 2019Updated Aug 3, 2026
Labelsenhancement

There are quite a few features currently in serde implementations where in-band signalling is used to communicate data from the deserializer into deserialize. This is turning into quite a big problem over time because it destroys the interoperability of serde in various ways.

A good example is the arbitrary_precision flag in serde_json. Since serde has no notion of a large integer the deserializer will deserialize numbers internally by emitting an object with a special $serde_json::private::Number key which is understood by serde_json itself. This however means that if someone tries to use this with other value types (like many libraries have) it will fail.

We use two value types in our own code: we have a fork of serde's internal Content type as well as a variation of serde_json's Value type that carries additional annotations. Currently if we turn on arbitrary_precision as feature we break everything because now any number is emitted as $serde_json::private::Number which none of our deserializers support. However we also feel very uncomfortable turning on this feature because even if we fix our own stuff is not now has a chance to break more.

I understand that a correct fix for this is tricky but I do wonder if it makes sense to start extending the underlying serde type system to carry some extension data around because I think this is going to eventually bite us much harder. I'm not entirely what the best solution is but I think extending the list of serde types to carry some internal extension value is most likely going to play a role in it. Similarly to how u128 was retrofitted we could add a new Signal value that formats can add extra information to.

Still not sure how to best make interoperability work best. Most likely deserializers should be encouraged to invoke some sort of default behavior when they find something like that.