Fidelity notes

This package reconciles and ports research code, and faithful sometimes means reproducing behaviour that looks wrong. This page is the complete record of those decisions — what is reproduced, what is switchable, and where this implementation deliberately differs — for anyone reproducing published work or comparing numbers across implementations. A newcomer can skip it entirely; nothing here is needed to use the package.

The reference behaviours kept behind switches

The ported algorithms default to the original arithmetic, because the published baselines were produced by exactly that arithmetic. Each behaviour is named, switchable, and defaulted to the original:

Default	What it means
gae="degenerate"	Advantage estimation collapses to r − V; gamma and lam have no effect. episodic makes them live.
ratio="sequence_mean"	The importance ratio uses the mean per-token log-probability, so the clip constrains average drift rather than each token.
action_tokens="qwen_think"	The policy gradient is gated on <think> literals and JSON-opening tokens — a rule specific to one model family. all lifts it.
old_policy_pass="train"	Dropout is active during the no-grad behaviour pass, so the ratio is not 1.0 even on the first minibatch. This one looks unintended in the original; it is reproduced because the baselines carry it.
parameter_scope="lora_only" (EWC)	Matches PEFT's naming. A model without adapters matches nothing — and that case raises rather than silently training as plain PPO behind a healthy log line.

One default deliberately differs: fisher_dropout="disabled". The original disables every nn.Dropout module and calls the estimate deterministic, but attention dropout under scaled-dot-product attention is not such a module, so two estimates over identical inputs differ substantially. Reproducing an irreproducible number buys nothing; fisher_dropout="reference" restores the original handling.

Hindsight replay, as it actually is

There is no goal representation and no goal-conditioned reward: the mechanic is a max-clamp of the terminal reward against a constant, and success_reward defaults to a value calibrated to MORPHEUS's reward scale — against any other scale it is an arbitrary number. Replay draws len(batch) // 4 rows, so a window under four steps fills the buffer and never draws from it; a replay-mixed batch carries no episode lengths, which degenerates advantages from the second update onward whatever gae is set to; and the original samples from the global RNG and seeds it nowhere, which is why replay_seed exists here and is unset by default. EWC's Fisher draw is stratified by reward band — 40% of the budget to solved episodes, the remainder filled from the whole pool with duplication when the pool is small — so the estimate is reward-weighted, not uniform.

LCM's latent, and what a checkpoint does not carry

ppo_lcm infers a latent configuration and trains it with a consistency loss. In the original that latent reaches the policy exactly one way — a [REGIME: …] prefix on the prompt — and here the backend exposes the prefix (observe / regime_prefix) rather than injecting it, because the backend does not own the prompt. Unwired, ppo_lcm is PPO plus an auxiliary loss that changes no decision. Its regime head is untrained online, as in the original, which fits it in an offline warm-up; without a warm-up checkpoint the prefix's probabilities are near-uniform. Relatedly, a checkpoint carries the policy and not the algorithm's memory: a reloaded EWC policy has no consolidation state and a reloaded LCM policy has no context encoder.

The three reward stacks

The research code contains three reward implementations that disagree, and all three ship as named specifications, each differentially verified against its original: paper (the appendix's stated formula), experiment (the composite the training runs actually optimised), and eval8 (the eight-term ticket reward the online evaluation stepped with), with verification_progress as the standalone pass-fraction signal. Which of them produced the paper's published figures has not been confirmed; the default (paper) is a choice, recorded as one, not a finding.

Carrying numbers over from the research code

The research code contains an earlier implementation of the metric protocol under the same names, and it measures different things. Comparing against numbers it produced:

• Forgetting has the opposite sign — here, positive means the policy got worse on the revisit; there, positive meant it improved. Neither number looks wrong on its own, so this is the one to check first.
• Adaptation was anchored to the policy's own pre-shift rolling mean — it said a policy improved on itself, not whether it ended anywhere good. Here the segment anchor references the interval's own peak (a different self-relative question, with a defined cap at the interval length), and the ceiling anchor references a derived bound.
• Recovery used a variance criterion — reward variance under a threshold — where here it is distance to a settled level; a policy can be steadily bad. Stability was a standard deviation over a trailing window, not a variance over the occurrence.
• The two disagreed about whether a shift-anchored window stops at its configuration's end; both behaviours exist here (AdaptationWindow), with the protocol's own choice as the default.

Two ways to read "the final 20%"

The performance gap and the settled reward describe the settled end of a configuration, and not always the same end: the gap rounds the tail length and measures the first occurrence (after adaptation has had its chance); the settled reward takes the ceiling of the length and averages every occurrence. On a ten-step interval they agree; on a six-step one they measure one and two steps — so subtracting the settled reward from the bound will not reproduce the gap. Both are faithful ports of the definitions they came from, unchanged.

The ceiling's correction, and the anchor's history

The derived ceiling reproduces both figures in the research record exactly — including a correction the original authors worked out by hand after finding the shipped default roughly three times too high. That episode is also why the default anchor moved: anchored to a ceiling no policy approaches, adaptation and recovery stop discriminating — the original analysis saw four algorithm families all report the interval-length cap — so the default became each segment's own behaviour, with the ceiling formulation retained for rewards whose ceiling is attainable. Under the ceiling anchor's own defaults, recovery and adaptation additionally ask nearly the same question (any mean below 1.5·R satisfies both), which is why published tables show a recovery of exactly 1 wherever adaptation was fast.