RFC: Index-based progress tracking to reduce context load for long-running tasks

Background

Ralph is an autonomous AI agent loop that implements features story-by-story, maintaining context across iterations via progress.txt. For long-running tasks this file grows unboundedly — every new story must load the entire history even when only a small slice is relevant.

Problem

progress.txt is append-only and flat. On each new user story, Ralph loads the full file to get context. On large projects this means loading thousands of lines of history to find a handful of relevant patterns, wasting tokens and increasing latency.

Proposed Solution: Indexed Progress Table

Maintain a lightweight index table at the top of progress.txt that maps topics to small reference files stored in a ralph/references/ folder.

Index format (top of progress.txt)

## Progress Index

| Topic | Reference Path |
|---|---|
| Auth flow learnings | ralph/references/auth-flow.md |
| DB schema decisions | ralph/references/db-schema.md |
| API error handling patterns | ralph/references/api-errors.md |

Rules for the agent

When completing a story:

1. If a reusable pattern or learning emerged, write it to ralph/references/<topic>.md
2. Add or update its row in the index table at the top of progress.txt
3. Append the normal one-line narrative summary to the log as usual

When starting a story:

1. Read only the index table (always tiny — a few KB regardless of project size)
2. Identify which reference files are relevant to this story's acceptance criteria
3. Read only those files — skip the rest

CLAUDE.md addition (minimal)

## Context Indexing

Maintain a progress index table at the top of progress.txt:
- Column 1: Topic (short human-readable name)
- Column 2: Path to ralph/references/<file>.md

When completing a story:
1. If a reusable pattern/learning emerged, write it to ralph/references/<topic>.md
2. Add or update its row in the index table
3. Append a one-line summary to the narrative log as usual

When starting a story:
1. Read ONLY the index table
2. Identify which reference files are relevant to this story's acceptance criteria
3. Read only those files — skip the rest

Why This Works

The index stays small forever regardless of project length. Agents already have the judgment to decide what's "relevant" — giving them a structured table makes that judgment cheap (scan 10 rows, read one file) instead of expensive (load the full history).

This also complements the existing CLAUDE.md-per-directory pattern: directory-level CLAUDE.md files capture spatial knowledge (what lives where), while the references/ index captures temporal knowledge (what was learned across stories).

Relation to Existing Work

This is in the spirit of the token-efficiency RFCs already open (the Harness Mode discussions) but targets a different layer: the persistent context store rather than the agent loop architecture. The two are complementary and independent.

Implementation

The change is purely additive — a small addition to CLAUDE.md and an optional ralph/references/ folder. Projects that don't adopt it behave exactly as today.

Happy to open a PR if there's interest.