Design: ThinkingBox x Agent Lightning stateful MCP training example

Author: hamedrabahCreated Aug 25, 2026Updated Sep 4, 2026

Ownership question

Would maintainers prefer this integration to live in:

  1. microsoft/agent-lightning as examples/thinkingbox/;
  2. microsoft/thinkingbox as an Agent Lightning recipe; or
  3. a small split across both repositories?

Proposed thin slice

ThinkingBox already exposes executable scalar rewards through TestResult.reward and identifies decode/test infrastructure errors with is_system_error. Agent Lightning already supplies a rollout-scoped OpenAI-compatible endpoint plus explicit succeeded/failed rollout states.

I propose one bundled cloud_drive example that:

  • points ThinkingBox's existing AOAISession(endpoint_url=...) at $AGL_OPENAI_BASE_URL/chat/completions;
  • emits an Agent Lightning reward only for a valid TestResult;
  • treats decode errors, test-system errors, timeouts, and exhausted gateway failures as failed rollouts, never as zero reward;
  • includes a credential-free mocked smoke test;
  • reports pass@1, all-k reliability (pass^k), and consecutive-run/long-horizon reliability;
  • documents local execution first, with Kubernetes and the full ThinkingBox dataset as optional follow-ups.

This should require no ThinkingBox core change for the initial version.

Acceptance criteria

  • The mocked path covers gateway -> ThinkingBox agent -> MCP tool -> executable test -> reward event.
  • A valid assertion failure can produce reward 0; infrastructure failures produce a failed rollout and no reward event.
  • The local recipe runs the bundled scenario without external data or credentials.

If Agent Lightning should own this, I can send a focused PR with the example, tests, and documentation.

Source: microsoft/agent-lightning