Structured experiments on G1h 2.9B: L_state trajectory reg, N/K/mode effort frontier, causal state probe, multi-slot capacity

Hi Peng,

I've spent the last few weeks running structured experiments on G1h 2.9B. Several results I think are worth your time.

1. State-trajectory regularization (L_state)

I added a training loss that rewards WKV state motion rather than suppressing it — sign-flipped SFA, applied per-layer at the positions A0.5 probing identified as load-bearing (L4, L16, L20, weighted by zero-layer KL). Combined with an ε-mask: full weight inside <think> spans, 5% outside. The idea is that reasoning tokens should produce large, curved state trajectories; answer tokens should produce clean emission.

Results on a 48-task reasoning benchmark (num_predict=512):

The fine-tune shifts the profile: symbolic 25%→75%, scheduling 16.7%→50%. Bit_decoding regressed (18.8%→0%) — explained below.

Training lineage. The 43.8% checkpoint is the result of four sequential LoRA fine-tunes on G1h 2.9B base:

Steps 7+8 account for ~97M tokens; steps 9+9b for ~320k. The ε-mask result is notable precisely because 71k RFC tokens sufficed to restore extraction while keeping the action-chain state-motion properties intact.

Note: tasks.jsonl was Claude-authored, so scores are upper-bound estimates until an independent algorithmic eval generator is built.

2. N/K/mode effort frontier — three dials, N=3 collapse

Industry effort controls expose a single scalar — CoT-token budget. RWKV exposes three orthogonal test-time compute axes that transformers cannot:

• N — state-refinement passes: same prompt fed through the backbone N times before decoding. Each pass runs WKV recurrence on the accumulated state — not a reset. No tokens emitted.
• K — intermediate token budget: K tokens decoded and re-fed into WKV, updating state at each step. Invisible to the user.
• mode — what drives K: silent (K=0), prompt_cot (prompt continuation), state_readout (free decode from refined state — the RWKV-native analogue of hidden thinking).

The key architectural point: for Transformers, CoT tokens are read by attention. Human-readable reasoning ("Step 1: …") works because attention indexes over token content. For RWKV-7, there is no attention over intermediate tokens — each token updates state s(t+1) = f(s(t), x(t)) and surface form is irrelevant. Training on human CoT traces therefore constrains intermediate tokens to a subspace optimised for readability rather than WKV state-update quality. The N axis (pure state refinement, no token emission) is the cleanest test: requires no specialised training, tests raw WKV accumulation directly.

Empirical results (N-sweep, step8 checkpoint):

• N=1: 27.1%
• N=2: 33.3% (best)
• N=3: 6.3% — catastrophic collapse (−27pp)

N=2 consistently beats single-pass across checkpoints. N=3 always collapses. Proposed mechanism: DSL-format training creates an implicit <tool_call> attractor. By pass 3, WKV state has accumulated "waiting for tool_result" and the reasoning signal is gone. The model has no gating criterion for when accumulation is done.

One bit_decoding task was solved: bit_sub_01 at N=2, K=128, state_readout mode. The state had accumulated bit-position bookkeeping across both passes. This is the only bit_decoding solve in the full sweep (0/16 otherwise) — the task is a compute-depth problem, not a knowledge gap.

Full framing and compute cost model: docs/effort-frontier.md.

3. H8/H9 confirmed — G1h amplifies state computation ~40×

I ran a causal probe (1152 paired forwards, 4 corruption types) on both 0.4B and 2.9B:

• σ-slopes: G1h 1.58–1.67 (superlinear) vs World baseline 1.13–1.19
• Cross-prompt / noise ratio: 34–40× (state carries prompt-conditional structure)
• Layer hotspots: L4, L16, L20 — consistent across both scales
• G1h gauss@σ=0.1 KL: 40× higher than World (0.23 vs 0.006)

The G1 training recipe amplifies what the WKV state does computationally, not just what it outputs. These layer hotspots are what determined the L_state weighting (w₁₆=0.506, w₂₀=0.238, w₄=0.035).

4. Multi-slot working memory — base capability and fragility

G1h 2.9B base holds 8 parallel colour-name slots at 53% accuracy (P=1 retrieval, no fine-tuning). G1d 0.4B base: ~11%. The 5× gap is emergent from G1 pre-training. It degrades properly at depth: G1h K=8, P=2 drops to 21% — not a retrieval artifact. Action-chain fine-tuning (step7) hurt this: K=2, P=1 dropped from 65% to 25%. The model specialised to one-shot slot access and lost depth retention. The capacity is genuine and fragile.

5. ROSA + WKV: additive sum, effort frontier angle

There's community disagreement on whether ROSA in RWKV-8 replaces WKV or runs alongside it. From reading the RWKV-LM 3.2-rwkv-v8-with-rosa branch: it's additive — ROSA_QKV_B_1bit is a separate block, output combined with RWKV_Tmix_x070:

x = x + x_wkv + x_rosa   # not x = x + x_rosa

I built an additive probe (g1h_rosa_block.py: zero-init output projection, forward hook on each layer) to test whether G1h could learn to use both channels simultaneously. Deferred: the reference ROSA implementation runs at ~1 tok/s on CPU making eval infeasible, and the one ROSA checkpoint I had (BlinkDL/rwkv-8-pile, ~1.25B) uses a different vocabulary — loading with strict=False leaves most weights random, making results uninterpretable.

The interesting question for the effort frontier: if ROSA handles algorithmic-precision tasks (exact bit manipulation, digit arithmetic, suffix matching) via the suffix automaton, and WKV handles semantic accumulation, RWKV-8 would have a natural 4th dial — how much of the compute budget to route through WKV vs ROSA per query. A scheduling or reasoning task would want mostly WKV; a bit-decoding or exact-copy task would want mostly ROSA. Whether the gating is trainable or fixed at inference time is an open design question. Once production-scale RWKV-8 weights exist, this seems worth probing.

6. Misc probes

• H20 (aporia holding): G1d 0.4B maintains p(neither)=0.75 on 100 contradictory prompts. Modal collapse rate 0.54 — better than expected at this size.
• H21 (premise validity): G1h 2.9B WKV state separates valid from structurally invalid premises with test F1=1.00 (8/8 test items). G1d 0.4B: F1=0.75.
• H22 (attribution): LOO F1=0.947 on 240 labelled items — WKV state alone distinguishes attributable from unattributed claims. Held at scale.

Questions

1. L_state prior art: Is there anything in the RWKV ecosystem shaping WKV trajectory as a training objective? I've found nothing directly comparable — closest external cousin is SFA with inverted sign, applied to CNNs/RNNs, never WKV.

2. N=3 collapse: Is the collapse an architecture property (WKV state saturates / self-interferes after 2 passes) or a training artifact of DSL-format corpus? If WKV state capacity is bounded in a principled way, there should be a natural N* — do you have intuition here?

3. G1h base bit_decoding (18.8%): The base solves ~3/16 bit-decoding tasks without fine-tuning. Gemma3 manages 6.3%, Qwen2.5 0%. Is this from pre-training data or an emergent effect of the G1 state-shaping recipe? The fine-tuned version dropped to 0% — I attribute this to format collision (DSL training suppressed multi-step bitwise CoT), not forgetting.

4. RWKV-8 hybrid: Once RWKV-8 lands at reasoning scale, is the ROSA+WKV additive sum trainable with a frozen WKV base (ROSA-only gradient), or does useful ROSA behaviour require joint training from scratch?

Checkpoint: Vaniello/noesis-rwkv7-g1h-2.9b (step9b, 39.6% — the peak 43.8% was lost to disk overflow). Full log + code + hypothesis tree: github.com/Vaniell0/noesis

Thanks for RWKV. The persistent-state substrate enables experiments that don't have a transformer analogue.

— Vaniello