from_cbor()/from_msgpack() do not validate UTF-8 in text strings at decode time (only dump() does)
Description
from_cbor() (and, by the same code path, from_msgpack()) decodes a CBOR/MessagePack text string by copying its raw bytes into a string_t without validating that they are well-formed UTF-8. RFC 8949 §3.1 requires CBOR major type 3 (text string) to contain "a UTF-8 string", and rejecting ill-formed UTF-8 there is expected decoder behavior — but this library defers that check to dump(), so from_cbor() itself neither throws nor returns a discarded value for invalid UTF-8 payloads. The invalid bytes only surface as an error later, if and when the resulting value is serialized back to JSON text.
Reproduction
#include <nlohmann/json.hpp>
#include <iostream>
using json = nlohmann::json;
int main() {
// CBOR: 0x62 = text string, length 2; payload 0xC0 0xAE is not valid UTF-8
std::vector<uint8_t> raw = {0x62, 0xc0, 0xae};
auto got = json::from_cbor(raw, /*strict*/true, /*allow_exceptions*/false);
std::cout << "is_discarded: " << got.is_discarded() << "\n"; // false
std::cout << "type: " << got.type_name() << "\n"; // string
got.dump(); // throws json.exception.type_error.316:
// "invalid UTF-8 byte at index 0: 0xC0"
}
So from_cbor() reports success (a normal, non-discarded string value) for input the CBOR spec requires to reject; the error only appears later and only if the caller happens to call dump() (or another UTF-8-sensitive operation) on the resulting value.
Traced to binary_reader::get_string(), which forwards straight to get_bytes() — a raw byte copy with no UTF-8 check. Contrast with the text/JSON lexer, which does validate UTF-8 during scanning (scan_string() in lexer.hpp), and with dump()'s own validation (type_error.316).
Why this seems worth a look
docs/features/binary_formats/bson.mddocuments comparable BSON leniency explicitly, with its own "Lenient BSON input handling" admonition and an escape hatch ("validate it separately before passing it tofrom_bson()"). I could not find an equivalent note for CBOR/MessagePack on the CBOR docs page or thefrom_cborAPI docs — so this specific deferred-validation behavior currently isn't documented anywhere I could find.- It means
from_cbor(..., /*allow_exceptions=*/false)— the pattern used specifically to avoid exceptions and get adiscardedsentinel on bad input instead — does not actually catch this category of malformed input; the exception can still surface later, from an unrelated call (dump()), if the caller isn't also validating UTF-8 there.
Suggested fix / alternatives
Either (a) validate UTF-8 in get_string()/get_bytes() at CBOR/MessagePack decode time, consistent with the text-JSON lexer, or (b) document the deferred-validation behavior the way BSON's leniency is documented, so it's a known, intentional tradeoff rather than a surprise.
Context
Found while investigating a third-party compliance report (#5525) — most of that report's findings turned out to be a bug in the reporting tool itself, but this particular case survived verification against the current develop header (commit 3bfe2b6da7393af5cfd68c44f58a1058e60499e2).
Source: nlohmann/json