`ConfigStd.Unmarshal` silently accepts invalid JSON string escapes where `encoding/json` returns an error

Author: brualanCreated May 10, 2026Updated May 10, 2026

Describe the bug

sonic.ConfigStd.Unmarshal accepts JSON strings containing invalid escape sequences (e.g. \0, \\\x, \v, \1) without returning an error. encoding/json correctly rejects the same input with, for \0:

invalid character '0' in string escape code

This is a strict violation of RFC 8259 §7. The ABNF grammar defines the only valid characters after an escape (\, %x5C) as:

Sequence	Meaning
`\"`	quotation mark
`\\`	reverse solidus
`\/`	solidus
`\b`	backspace
`\f`	form feed
`\n`	line feed
`\r`	carriage return
`\t`	tab
`\\\uXXXX`	Unicode code point

None of \0, \\\x, \v, \1, etc. matches any of these productions. There is no ambiguity, no implementation-defined behaviour, and no optional leniency in the spec, because they are hard parse errors.

sonic.ConfigStd is documented as a drop-in replacement for encoding/json and is configured with ValidateString: true, making silent acceptance here doubly unexpected.

The bug is specific to the skip-value path: when unmarshaling into a struct and the JSON key has no matching field, sonic skips the value with a SIMD-accelerated string scanner that locates the closing quote but does not validate the content of escape sequences. The escape-content check only runs in the scalar fallback used for short strings, so the bug surfaces once the string is long enough for SIMD to take over. Empirically, ~29 bytes of remaining string content, and it persists at all greater lengths. Matched struct fields and interface{} targets go through different paths and are handled correctly.

The relevant code is advance_string_validate in native/scanning.h: the 32-byte and 64-byte SIMD chunks return as soon as the closing quote is found, while escape-sequence validation (advance_escape_validate) is only invoked from the scalar tail loop.

To Reproduce

package main

import (
        stdjson "encoding/json"
        "fmt"

        "github.com/bytedance/sonic"
)

// S has no field matching key "" — sonic takes its skip-value fast path.
type S struct{}

// Each input contains a different invalid escape (\0, \1, \v) followed by enough
// padding to push the string past sonic's SIMD threshold (~29 bytes after `\`).
// The padding is what triggers the bug: shorter strings fall back to the scalar
// validator, which correctly rejects the escape.
var inputs = [][]byte{
        []byte(`{"":"\0` + strings.Repeat("p", 28) + `"}`), // \0
        []byte(`{"":"\1` + strings.Repeat("p", 28) + `"}`), // \1
        []byte(`{"":"\v` + strings.Repeat("p", 28) + `"}`), // \v
}

func main() {
        for _, input := range inputs {
                var std, son S
                fmt.Printf("input=%s\n", input)
                fmt.Printf("  encoding/json:  %v\n", stdjson.Unmarshal(input, &std))
                fmt.Printf("  sonic.ConfigStd: %v\n", sonic.ConfigStd.Unmarshal(input, &son))
        }
}

Output:

input={"":"\0pppppppppppppppppppppppppppp"}
  encoding/json:  invalid character '0' in string escape code
  sonic.ConfigStd: <nil>
input={"":"\1pppppppppppppppppppppppppppp"}
  encoding/json:  invalid character '1' in string escape code
  sonic.ConfigStd: <nil>
input={"":"\vpppppppppppppppppppppppppppp"}
  encoding/json:  invalid character 'v' in string escape code
  sonic.ConfigStd: <nil>

Expected behavior

sonic.ConfigStd.Unmarshal returns a non-nil error for any input containing an invalid escape sequence, matching encoding/json.

Actual behaviour

sonic returns nil and silently ignores the malformed escape, as shown in the output above.

Sonic version:

v1.15.1

Environment:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/brualan/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/brualan/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/nix-shell-66243-0/go-build1474066616=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/tmp/tmp.59XRHDnlbj/go.mod'
GOMODCACHE='/home/brualan/go/pkg/mod'
GOOS='linux'
GOPATH='/home/brualan/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/0hfiiapvcwhdxc30m6606qx79nrik7jz-go-1.25.5/share/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/brualan/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/0hfiiapvcwhdxc30m6606qx79nrik7jz-go-1.25.5/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.5'
GOWORK=''
PKG_CONFIG='pkg-config'

Source: bytedance/sonic

View original on GitHub View discussion on GitHub