[Bug]: Direct-only cache response serialization races with core cleanup and can panic

Author: TheRobotCarlsonCreated Sep 16, 2026Updated Sep 22, 2026

Version

Bifrost transports v2.1.1, semanticcache v1.6.2, core v1.8.6
Go 1.27.0, Linux amd64
Direct-only caching: dimension=1, no embedding provider or model

Problem

PostLLMHook launches a background goroutine that retains the caller-owned *schemas.BifrostResponse. addNonStreamingResponse later calls json.Marshal on that pointer. Core clears ExtraFields.RawRequest/RawResponse after RunPostLLMHooks returns, so the cleanup races with the cache serializer.

We observed a gateway process panic followed by a restart. An in-flight chat request failed during that interval. The relevant stack, without request data:

panic: reflect: call of reflect.Value.Set on zero Value
reflect.Value.Set
encoding/json/v2.makeInterfaceArshaler.func1
encoding/json/v2.Marshal
github.com/maximhq/bifrost/plugins/semanticcache.(*Plugin).addNonStreamingResponse
    plugins/[email protected]/utils.go:461
github.com/maximhq/bifrost/plugins/semanticcache.(*Plugin).PostLLMHook.func2
    plugins/[email protected]/main.go:642

Minimal reproduction

Save the two files below in the same directory. Requires Bash, GNU timeout, and Go with automatic toolchain downloads enabled. Run:

bash

bash run.sh

The script downloads the released module and runs one self-contained in-package test with -race. It uses a minimal in-memory store, a synthetic response, and the same post-hook raw-field cleanup as core. No gateway, Redis, provider calls, credentials, or production payloads are required. Upstream's external-store cleanup TestMain is explicitly excluded.

Expected: the asynchronous cache write owns a stable response snapshot. Actual: the race detector reports the JSON reader against the raw-field cleanup; the cache snapshot also loses the raw response. The exact reflection panic is timing-dependent and is not claimed as deterministic in this MRE.

Suggested fix

Snapshot/serialize before PostLLMHook returns, and pass only owned data to the asynchronous cache writer. Apply equivalent ownership protection to streamed responses retained by the accumulator. Cache serialization failures should remain nonfatal to the successful provider response. The race exists in direct-only mode because this plugin implements exact-response caching as well as semantic matching.

Locally, the matching unary/streaming regression fails against the release and passes under -race after this ownership fix. The regression also checks that an unserializable response skips the cache without failing the provider response.

The standalone MRE fails on the released module with a race report and snapshot assertion, and passes ten repetitions against the locally patched source. This reproduction does not load our custom plugin or application code.

run.sh

bash

#!/usr/bin/env bash
set -euo pipefail
repro_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export GOWORK=off GOTOOLCHAIN=go1.27.0
scratch_dir="$(mktemp -d)"
trap 'rm -rf "$scratch_dir"' EXIT

# Fetch the released module, not a fork or a modified checkout.
timeout 120s go mod download github.com/maximhq/bifrost/plugins/[email protected]
module_dir="$(go env GOMODCACHE)/github.com/maximhq/bifrost/plugins/[email protected]"
cp -R "$module_dir/." "$scratch_dir/"
chmod -R u+w "$scratch_dir"
cp "$repro_dir/responseownership_test.go" "$scratch_dir/"
cd "$scratch_dir"

# Do not run upstream TestMain: it probes/deletes external test namespaces.
# Compile production files and this single self-contained test only.
sources=()
for file in *.go; do
  [[ "$file" == *_test.go ]] || sources+=("$file")
done
timeout 300s go test -race -run '^TestCacheResponseOwnership$' -count=1 -timeout=30s "${sources[@]}" responseownership_test.go

responseownership_test.go

package semanticcache

import (
	"context"
	"encoding/json"
	"runtime"
	"strings"
	"testing"
	"time"

	bifrost "github.com/maximhq/bifrost/core"
	"github.com/maximhq/bifrost/core/schemas"
	"github.com/maximhq/bifrost/framework/vectorstore"
)

// Only these two store methods are used by the post-hook's direct-cache path.
// No Redis, embedding model, network request, or credentials are involved.
type captureStore struct {
	vectorstore.VectorStore
	payload string
}

func (*captureStore) RequiresVectors() bool { return false }
func (s *captureStore) Add(_ context.Context, _, _ string, _ []float32, metadata map[string]interface{}) error {
	s.payload = metadata["response"].(string)
	return nil
}

func TestCacheResponseOwnership(t *testing.T) {
	previous := runtime.GOMAXPROCS(1)
	defer runtime.GOMAXPROCS(previous)
	store := &captureStore{}
	p := &Plugin{
		store:  store,
		config: &Config{DefaultCacheKey: "fixture", Dimension: 1, TTL: time.Minute},
		logger: bifrost.NewDefaultLogger(schemas.LogLevelError),
	}
	ctx := schemas.NewBifrostContext(context.Background(), time.Now().Add(time.Second))
	ctx.SetValue(schemas.BifrostContextKeyRequestID, "fixture-request")
	ctx.SetValue(CacheTypeKey, CacheTypeDirect)
	p.createCacheState("fixture-request").ParamsHash = "fixture-params"
	res := &schemas.BifrostResponse{ChatResponse: &schemas.BifrostChatResponse{
		ExtraFields: schemas.BifrostResponseExtraFields{
			RequestType: schemas.ChatCompletionRequest,
			RawResponse: json.RawMessage(`{"synthetic":true}`),
		},
	}}
	if _, _, err := p.PostLLMHook(ctx, res, nil); err != nil {
		t.Fatal(err)
	}
	// Core performs this cleanup after RunPostLLMHooks returns (core/bifrost.go).
	res.ChatResponse.ExtraFields.RawResponse = nil
	p.writersWg.Wait()
	if !strings.Contains(store.payload, `"raw_response":{"synthetic":true}`) {
		t.Fatal("async cache writer read the caller's response after raw-field cleanup")
	}
}

Source: maximhq/bifrost

View original on GitHub View discussion on GitHub