#751 Cohort construction is non-deterministic: the same input builds a different persona population on every run

摘要：同一份输入重复跑一遍，每次都会构建出不同的人群——本体、实体、人格全是新的。交互式使用没问题，但只要涉及多次运行之间的比较，结果就不可复现，而 API 层面看不出这一点。

我们实测：同一输入、逐字相同的访谈提示词，两次跑出 9.5% 和 60.0% 的弃权率。把下列三处温度设为 0 之后，四次运行中有三次产出逐字节相同的人格花名册。

建议：把这几处做成可配置，默认值保持现状（如 MIROFISH_COHORT_TEMPERATURE，不设＝现有行为，设 0＝可复现人群），对现有用户零影响。详情见下方英文全文。

Thanks for MiroFish — being able to run the whole stack against a local model is what made the measurement below possible.

Issue: re-running the same input builds a different crowd each time — new ontology, new entities, new personas. For interactive use that is fine and arguably desirable. For any use where runs are compared against each other it makes results non-reproducible, and nothing in the API surface hints at it.

What we measured

Running the same event description repeatedly through graph/ontology/generate → graph/build → simulation/prepare → simulation/start → simulation/interview, with a byte-identical interview prompt, gave 9.5% and 60.0% abstention on two runs of the same input.

After setting the temperatures below to 0, three of four subsequent runs produced byte-identical persona rosters (comparing reddit_profiles.json, ignoring the random numeric suffix on usernames); the fourth was the first run after a container start and differed by a single entity.

Full data, hashes and the reproduction commands: https://github.com/outerheaven666/llm-agent-simulation-variance/blob/v1.0-finding/README.md

The call sites

file	line	current
`backend/app/services/ontology_generator.py`	~181	`temperature=0.3`
`backend/app/services/oasis_profile_generator.py`	~520	`temperature=0.7 - (attempt * 0.1)`
`backend/app/services/simulation_config_generator.py`	~445	`temperature=0.7 - (attempt * 0.1)`

The third is reached via SimulationManager.prepare_simulation() and is easy to miss — we missed it on a first pass that only grepped the files we already knew were on the path.

We deliberately did not touch the interview stage. Per-agent response variation is the interesting part; only cohort construction needs pinning.

Suggested change

Make these configurable, with today's values as the defaults. Something like MIROFISH_COHORT_TEMPERATURE — unset gives current behaviour, 0 gives reproducible cohorts. Nothing changes for existing users.

We have a patch that hard-sets 0, but that is right for our use case and wrong as a default for yours, so it is not offered as-is. Happy to send a PR in whichever shape you prefer.

One implementation detail worth knowing first: both retry ladders read temperature = 0.7 - (attempt * 0.1), i.e. 0.7 → 0.6 → 0.5 — descending, per the comment 每次重试降低温度. Anyone making the base configurable while keeping the ladder will find that a base of 0 computes negative temperatures on retry. Since the ladder's purpose is more reliable JSON on retry, and 0 is already the most reliable setting, a flat value is probably what is wanted when the base is 0.

Note for the Offline fork

nikmcfly/MiroFish-Offline adds a fourth site not present here — backend/app/storage/ner_extractor.py L85, temperature=0.1 — on the graph/build path. Happy to open a matching issue there if useful.

Also worth a README line (not a bug)

Both compose files declare the backend as build: . and mount only backend/uploads, so the Python source is baked into the image at build time. Editing the host tree changes nothing until docker compose build runs. This cost us a while, and one line in the docs would save the next person the same.

Context: this came out of a study measuring run-to-run variance of agent-population simulations at 7B scale on consumer hardware. The write-up, with the raw per-persona interviews and the roster hashes referred to above, is here:

https://github.com/outerheaven666/llm-agent-simulation-variance/blob/v1.0-finding/README.md

The finding — that readings of the same input are not reproducible at that scale even after the cohort is pinned — depends on the determinisation above being possible, which is what makes this fix load-bearing rather than cosmetic.

To be explicit: the finding is about what a 7B local model can support on 8 GB of consumer hardware, not a claim about MiroFish's design. The determinisation above is what let us rule out cohort churn as the explanation.