Internally tagged enum with a newtype variant holding `Vec<Self>` / `Box<Self>`: derived `Serialize` fails to compile with E0275 (and hangs rustc if the recursion limit is raised)
Summary
The enum-representations docs say an internally tagged newtype variant that holds a sequence fails at runtime. In practice the derived Serialize impl does not compile at all when the newtype variant holds Self (directly via Box<Self> or via Vec<Self>): rustc reports E0275 at the default recursion limit, and if the user follows the diagnostic and raises the limit, rustc spins for hours (rust-lang/rust#162356).
Reproducer
Same crate as in the linked rustc issue. Minimal shape:
#[derive(Serialize, Deserialize)]
#[serde(tag = "type")]
pub enum Expr {
Eq { field: String },
And(Vec<Expr>),
Not(Box<Expr>),
}
fn main() {
let _ = serde_json::to_string(&Expr::Eq { field: "a".into() }).unwrap();
}error[E0275]: overflow evaluating the requirement `&mut Vec<u8>: std::io::Write`
= note: required for `&mut serde_json::Serializer<&mut Vec<u8>>` to implement `_::_serde::Serializer`Deserialize alone compiles and works. serde 1.0.228 / 1.0.229, serde_json 1.0.150 / 1.0.151, rustc 1.94.1 and 1.98.1.
Cause
Serialize for Expr with serializer S serialises And(Vec<Expr>) through TaggedSerializer<S>, whose serialize_newtype_variant needs Expr: Serialize for TaggedSerializer<S>, and so on; the obligation nests a new serializer type at every level and can never be discharged.
Ask
One of:
serde_deriverejects this shape with a targeted error ("internally tagged newtype variant containingSelfcannot be serialized; use a struct variant"), instead of letting the trait solver produce E0275 or a hang.- Or the docs state that such variants do not compile, not merely that they fail at runtime, so users do not reach for
recursion_limit.
The workaround is struct variants (And { filters: Vec<Expr> }, Not { expr: Box<Expr> }), which compile under the default limit and round-trip.
Source: serde-rs/serde