Anatomy of a run

Reinforcement-learning vocabulary was built for episodic training, and in a persistent world its central word — episode — stops doing the work it used to do. This chapter defines the units a run here is actually made of. Three of them partition the same axis of steps in different ways, none of their boundaries coincide, and each answers to a different owner.

The four units

A step is one interaction: the agent acts, the environment responds with an observation and a reward. Steps are the shared axis everything below is measured on, and each one is recorded as one row of the run's trace.

A rollout window belongs to the harness. It is an administrative slice — window_steps of experience collected, handed to the algorithm for one update, followed by the next window. A window boundary is a training event and nothing else: it does not reset the environment, does not end an episode, and does not mark anything the evaluation layer can see. Choosing window_steps trades update frequency against the amount of experience each update sees.

An episode belongs to the environment. It ends only when the environment says so — terminated for a genuine terminal state, truncated for an administrative cutoff the environment itself imposes. A persistent world may never say so: a MORPHEUS world has no terminal state by design, so a run there is one episode that simply continues. CartPole, by contrast, terminates every few hundred steps at best. Both are legitimate, and the harness treats episode ends as data rather than as scheduling.

A configuration interval belongs to the evaluation protocol. It is a maximal stretch of steps recorded under one regime label — one set of world conditions. The metrics segment the run by these intervals and by nothing else: adaptation speed counts steps from the start of an interval, forgetting compares two intervals recorded under the same conditions, per-configuration reward averages within them. Window and episode boundaries are invisible to all of it.

A world instance sits above all three: one provisioned environment, living for the whole run. In a persistent world, reset() is not a rewind — it destroys the world and builds another, discarding everything the agent's actions accumulated. The harness therefore never resets on its own initiative. auto_reset exists for environments with genuine terminal states, where starting the next episode is the only way to continue, and it defaults to off because the one environment the wrong setting damages is the one where reset is destructive.

The rule that relates them

A window boundary ends nothing, and a configuration shift ends nothing. Only the environment ends an episode.

This is a deliberate property, not an accident of implementation, and one half of it is load-bearing for the science: a shift must not end the episode. If it did, the agent would be handed the boundary for free — episode start would announce "conditions just changed" — and adaptation would be trivial. Crossing a shift changes what the agent observes and how it is scored, and nothing else. The other half keeps training honest: windows are how the algorithm paces its updates, and letting them leak into the environment or the metrics would make results depend on a tuning knob.

One consequence worth knowing early

Rewards composed from a specification may include terms that pay only at an episode's end — a final ledger settlement, a completion bonus. Those terms fire once per episode, not once per window. A window shorter than the episode legitimately sees only the per-step terms, and the window containing the episode's last step sees the payout. Every layer downstream accounts for this: the upper bound distinguishes an ordinary step's ceiling from the final step's, and forgetting can exclude the payout so a comparison between two intervals is not decided by which of them happened to contain an episode's end.

The trace is the record

Every step becomes one row: the reward, its per-component breakdown when a specification produced it, the regime label and its origin, and the run's identity (run_id, seed, algorithm_id, task). Everything in this chapter is recoverable from those rows alone — episode ends from the payout terms, intervals from the label column, windows from nothing (they are deliberately not recorded, because no downstream consumer may depend on them). The evaluation layer reads traces and nothing else: no environment, no model, no deployment. That is what makes a run scoreable months later, on a machine that has none of the software that produced it.