#4899·baml

[bug] Playground Tests panel shows "No tests found" since 0.19.0: requestCollectTests returns an empty tree

Author: BenSpexCreated Sep 16, 2026Updated Sep 16, 2026

Product

VS Code Playground

Describe the bug

Since toolchain 0.19.0 the playground's Tests panel is empty for every project. It shows "No tests found", while baml test --list in the same project lists the tests and baml test runs them.

The server answers requestCollectTests with an empty tree (testCollectionResult data [], no expandError). The build is current (isBexCurrent: true), the $init_test_<file> function is present in the function list, and the server log shows no error. The same project on 0.18.1-nightly.20260909.a returns the expected tree.

Toolchain baml test --list testCollectionResult
0.18.0 2 tests root::adds + lazy testset root::group
0.18.1-nightly.20260909.a 2 tests root::adds + lazy testset root::group
0.19.0 2 tests []
0.19.0 with -F beta 2 tests []
0.19.1-nightly.20260915.a 2 tests []

So the regression landed between 0.18.1-nightly.20260909.a (released 2026-09-10) and 0.19.0 (released 2026-09-11).

We first saw it on a real project with 34 .baml files: the CLI lists 142 tests, the playground none. It happens both in the editor extension installed from the 0.19.0 toolchain's assets/baml-vscode.vsix and in the standalone baml playground, so it looks server side rather than UI side.

Reproduction Steps

Save both files below in one directory and run it with an installed toolchain version. The script creates a throwaway project with one top-level test and one testset, lists tests with the CLI, starts baml playground --no-open, and asks the server for the test tree over /api/ws, the same message the Tests panel sends. It needs Node 22 or newer for the global WebSocket.

bash
./repro.sh 0.18.1-nightly.20260909.a   # returns the tree
./repro.sh 0.19.0                      # returns []

Output on 0.18.1-nightly.20260909.a:

--- baml test --list
    Selected 2 test(s)
  root::adds  (<testset>)
  root::group::inner  (<testset>)
--- baml playground
toolchain 0.18.1-nightly.20260909.a protocol 2
build generation 1 test init functions 1
testCollectionResult [{"$type":"testing.SerializedTest","type":"test","name":"root::adds"},{"$type":"testing.SerializedTest","type":"lazyTestSet","name":"root::group"}]

Output on 0.19.0, and identically on 0.19.1-nightly.20260915.a:

--- baml test --list
    Selected 2 test(s)
  root::adds  (<testset>)
  root::group::inner  (<testset>)
--- baml playground
toolchain 0.19.0 protocol 2
build generation 1 test init functions 1
testCollectionResult []

To see it in the UI instead, open the printed project in baml playground on 0.19.0 and expand Tests. It shows "No tests found". "Re-collect tests" sends the same requestCollectTests message and gets the same empty tree.

repro.sh

bash
#!/usr/bin/env bash
# Usage: ./repro.sh <toolchain-version> [port]
# Creates a two-test project pinned to <toolchain-version>, lists tests with the CLI,
# then starts `baml playground` and asks it for the test tree over /api/ws.
set -euo pipefail
VERSION=$1
PORT=${2:-4291}
HERE=$(cd "$(dirname "$0")" && pwd)
DIR=$(mktemp -d)
trap 'kill "${PG_PID:-}" 2>/dev/null || true; rm -rf "$DIR"' EXIT
mkdir -p "$DIR/baml_src"
printf '[package]\nname = "collect-repro"\n\n[toolchain]\nversion = "%s"\n' "$VERSION" > "$DIR/baml.toml"
cat > "$DIR/baml_src/main.baml" <<'BAML'
function add(a: int, b: int) -> int {
    a + b
}

test "adds" {
    assert.equal(add(1, 2), 3)
}

testset "group" {
    test "inner" {
        assert.equal(add(2, 2), 4)
    }
}
BAML
cd "$DIR"
echo "--- baml test --list"
baml test --list
echo "--- baml playground"
baml playground --port "$PORT" --no-open > "$DIR/playground.log" 2>&1 &
PG_PID=$!
for _ in $(seq 1 60); do
  if (exec 3<>"/dev/tcp/127.0.0.1/$PORT") 2>/dev/null; then break; fi
  sleep 0.5
done
node "$HERE/collect-probe.mjs" "$PORT"

collect-probe.mjs

javascript
// Usage: node collect-probe.mjs <port>   (Node >= 22 for the global WebSocket)
// Connects to a running `baml playground`, asks for the test tree of every project
// once the build is current, and prints what the server sends back.
const port = process.argv[2] ?? '4265';
const ws = new WebSocket(`ws://127.0.0.1:${port}/api/ws`);
let projects = [];
let asked = false;
ws.onopen = () => ws.send(JSON.stringify({ type: 'requestState' }));
ws.onmessage = (ev) => {
  const m = JSON.parse(ev.data);
  if (m.type === 'hello') console.log('toolchain', m.toolchainVersion, 'protocol', m.playgroundProtocol);
  const n = m.notification;
  if (!n) return;
  if (n.type === 'listProjects') projects = n.projects.map((p) => p.path);
  if (n.type === 'updateProject' && n.update.isBexCurrent && !asked) {
    asked = true;
    const init = n.update.functions.filter((f) => f.name.startsWith('$init_test')).map((f) => f.name);
    console.log('build generation', n.update.generation, 'test init functions', init.length);
    for (const p of projects) ws.send(JSON.stringify({ type: 'requestCollectTests', project: p }));
  }
  if (n.type === 'testCollectionResult') {
    const tree = new TextDecoder().decode(new Uint8Array(n.data ?? []));
    console.log('testCollectionResult', tree, n.expandError ? `expandError=${JSON.stringify(n.expandError)}` : '');
    ws.close();
    process.exit(0);
  }
};
setTimeout(() => { console.log('timeout: no testCollectionResult'); process.exit(2); }, 60000);

Workaround: serve the playground with an older toolchain binary, for example ~/.baml/toolchains/0.18.1-nightly.20260909.a/bin/baml-cli playground, as long as the project still compiles there.

BAML Version

0.19.0 (also 0.19.1-nightly.20260915.a); baml wrapper 0.2.0

Language/Framework

Node.js

LLM Provider

Other

LLM Model

Not involved; the tests are pure BAML with no LLM call

Operating System

Linux

Browser

Chrome

Code Editor

Other (Antigravity 1.107, a VS Code fork, with the extension from the 0.19.0 toolchain VSIX), and the standalone baml playground