#2734·stagehand

Evals compiled runtime-path test fails on Windows due mixed path semantics

Author: abhinavkr26104Created Aug 15, 2026Updated Aug 15, 2026

Description

The compiled-caller branch of resolveRuntimeTasksRoot() deliberately returns a normalized forward-slash path, but its unit test builds the expected value with the host-specific path.join().

The two values agree on POSIX. On Windows, the implementation returns /repo/packages/evals/dist/esm/tasks while the assertion expects \repo\packages\evals\dist\esm\tasks, so the evals unit suite has a deterministic platform-only failure.

Code reference

  • packages/evals/runtimePaths.ts:111-118 normalizes the caller path and constructs the compiled tasks root with forward slashes.
  • packages/evals/tests/runtimePaths.test.ts:14-20 supplies POSIX fixture paths but uses host-native path.join() for the compiled expectation.
  • The source-mode case at packages/evals/tests/runtimePaths.test.ts:6-11 happens to use path.join() in both implementation and expectation, so it passes.

Reproduction

On Windows:

powershell
.\node_modules\.bin\vitest.CMD run packages/evals/tests/runtimePaths.test.ts

The uses compiled tasks for built CLI callers case fails with the equivalent of:

expected "\repo\packages\evals\dist\esm\tasks"
received "/repo/packages/evals/dist/esm/tasks"

Expected behavior

A fixture written as a POSIX path should have a platform-independent expected result, or both sides should be normalized by the same helper.

Actual behavior

The assertion changes with the host OS while the function's compiled branch intentionally returns normalized separators.

Why it matters

This makes the evals unit suite fail on Windows for test-fixture formatting rather than product behavior. It also makes full-suite failures noisier and can mask genuine eval runtime-path regressions.

Duplicate/history check

I searched open and closed issues and PRs for resolveRuntimeTasksRoot, runtime path tests, Windows separators, and the compiled tasks directory. The code came through the evals/v4 history, but I found no report or PR addressing this assertion.