pi-rsi on CASMI26: how a biased proxy stalled the search

pi-rsi is our automated research harness: a research agent picks the next experiment, a worker agent implements it in its own Git worktree, and the harness measures the result with a frozen evaluator. This note covers a four-day run on the Kaggle CASMI26 task (identifying molecules from MS/MS spectra), where the search stalled for 19 nodes, what caused it, and what we changed. The interactive score chart and search tree show every node; hover or tap a node for its summary. The code is on GitHub.

What happened

The run used a local benchmark with 449 held-out molecules and MRR@25 as the official score. GPT-6.1 Sol was the research lead and the worker; GPT-6 Luna structured decisions and maintained the research wiki. The score rose from the baseline 0.307 to 0.404 at node n014 on the second day. The next 19 nodes produced no new best.

Every score is a proxy

The real target is the hidden leaderboard. The study cannot see it, so it optimizes stand-ins, and each layer adds noise and bias:

• the local official validation stands in for the leaderboard;
• the 296-molecule development set, which workers may tune on, stands in for the official validation;
• the research lead's delivery rule stands in for the development set.

The outer layer was already visibly off: n007 scored 0.3998 locally but 0.228 on the Kaggle public leaderboard. The stall came from the inner layer.

What went wrong

From n008 on, the research lead told workers to ship a new component only if no class and no source got worse, on either training seed. One of these segments had 14 molecules, and segment standard errors were as large as the gains being chased. With k such checks, a real but small gain passes with probability near 0.5^k, and only changes that cannot reorder existing answers get through. n014 passed because it only appended generated structures after the database candidates.

So workers built components and shipped them switched off. 22 of 31 official evaluations measured an output identical to the parent's, each costing about 38 minutes of two-GPU time. One example: n016 improved the aggregate on both seeds (+0.004 and +0.009) and was switched off because one class fell by 0.006.

The rule came from our own playbook ("never give back a won segment"). The harness already forbade the lead to invent thresholds for official results, so the lead put the threshold on the development set instead and said so in writing. A fresh course-correction reviewer flagged the drift after 8 stalled nodes; the lead followed the correction for one node and then returned to the same gates. The audit also found harness bugs: a tuning budget counted log lines instead of trials and failed three valid nodes, and about 30% of wiki updates were rejected by a too-strict validator.

What we changed

The fixes live in the harness and its system prompts, not in notes for one run:

• The playbook optimizes the official aggregate. A segment can veto a change only when its loss exceeds about two paired standard errors, and a component ships switched on when its pooled development gain is positive.
• Workers declare what they deliver. A component delivered switched off is measured switched on, and a node that delivers no new component skips the official evaluation.
• A proxy ledger records development, forecast and official deltas per node; rsi observe feeds results on the real target, such as a leaderboard score, back to the research lead. The setup review must describe how the official level differs from the real target.
• Repeated forecast misses and proposals that re-enter a paused direction force a full review.

The change to the playbook took effect in the running study; the harness changes apply to the next run. Replaying seven real decisions through the new extractor recovered six forecasts (+0.0065 to +0.01, all realised as 0); a full review would have been forced right after n026.

First result

Under the new rule, n033 kept its component off because its pooled development gain was exactly zero. n034, a supervised query–reference identity verifier, shipped switched on and scored 0.4185, a gain of 0.0146 against a noise band of 0.0089: the first new best in 20 nodes.

This is one data point on a reused validation set, not independent confirmation, and the local score still does not transfer to the leaderboard. The next round adds PubChem-scale candidates to narrow that outer gap, and will be the first real test of the delivery declarations.

Explore

The score chart and search tree, the snapshot data and the proxy ledger document what is shown. The README section and the research-loop documentation describe the mechanisms. No competition code, data or labels are published; the CASMI26 task pack stays private while the competition runs.