reward model HTTP service has no authentication, defaults to binding all interfaces, and the training client consumes whatever the endpoint returns

Title

serve_rm.py /get_reward is unauthenticated with --host default 0.0.0.0; the PPO client trusts endpoint-supplied rewards and leaks prompts and completions to the endpoint

Summary

The remote reward model channel has no integrity or authentication on either end:

• openrlhf/cli/serve_rm.py exposes POST /get_reward with no authentication of any kind. The argparse default for --host is 0.0.0.0 (binds every interface); --port defaults to 5000. The shipped examples/scripts/serve_remote_rm.sh only adds --port.
• openrlhf/utils/agent.py SingleTurnAgentExecutor._fetch_rewards_via_http POSTs {"query": [...], "prompts": [...], "labels": [...]} to each configured URL over plain HTTP and returns the JSON body. The body's rewards and scores values flow into Experience.rewards and Experience.scores with no validation, signature, or binding to the request.

Consequences for anyone who can reach the endpoint or intercept the traffic (relevant to the documented multi-node ray deployments, where the reward service regularly runs on a different node):

1. Verdict injection: whoever answers on the URL controls the training signal. A malicious or squatted listener returns chosen rewards for every sample and steers the policy update. Because the scheme is plain HTTP with no auth, the same holds for an on-path attacker.
2. Data exfiltration: every request body contains the full prompts and the policy completions ("query"). An endpoint owner or MITM harvests the training corpus and model outputs.
3. In tool-using agentic rollouts, the policy itself can make outbound HTTP calls if the user-provided agent module gives it such tools; the reward endpoint then also becomes reachable from the evaluated side.

Affected version

Pinned commit 3c3be6234e0cb353e76bb8019947db9dfe99fca7 (tag v0.11.0, 2026-08-13). Files: openrlhf/cli/serve_rm.py (route at lines 128 to 135, host default at line 85), openrlhf/utils/agent.py (_fetch_rewards_via_http at lines 322 to 356, consumption at lines 284 to 297).

Steps to reproduce

Repro directory: poc_f2_reward_channel (poc.py plus REPRO.sh). Loopback only, no model inference:

• The pinned serve_rm.py argparse defaults are read programmatically from the file (AST): --host default "0.0.0.0", --port default 5000, no auth anywhere in the file.
• The real SingleTurnAgentExecutor._fetch_rewards_via_http is called against a loopback server standing in for an unprivileged peer that bound the reward port first. The server returns a chosen reward (0.9876543) plus an attacker-controlled extra_logs dict.
• The server also records the request it received.

Observed output (deterministic across two runs):

"B_verdict_injection": {"attacker_reward_value": 0.9876543, "client_accepted_rewards": 0.9876543, "accepted": true}
"C_data_exfiltrated_to_endpoint_owner": {"keys_seen_by_server": ["labels", "prompts", "query"],
  "query_leaks_prompt_and_response": true}

Impact

Any actor with network reach to the reward service (or on-path between trainer and service) controls the reward signal of an RLHF run and reads its prompts and completions. Default exposure is widened by --host 0.0.0.0. Severity depends on deployment: loopback single-host deployments are exposed to any local process; multi-node ray deployments are exposed to the cluster network and whatever routes between nodes.

Suggested fix

• Default --host to 127.0.0.1 and require an explicit flag to bind wider.
• Add a shared-secret or mutual-TLS option to both serve_rm.py and _fetch_rewards_via_http, and validate response shape (finite floats, count matches the request) before consumption.
• Document the trust assumption of the reward channel in the multi-node guides.
• Consider hashing request and response into a run log so silent reward divergence is detectable after the fact.

Classification notes

Second executed instance of the "unauthenticated evaluation control plane" genus first minted from AgentBench (R03E1-M2, filed against THUDM/AgentBench for the agentrl worker API). Here the control plane is the reward channel itself rather than a results API. PUBLIC-READY: standard hardening advisory, requires a network or local-process position.