Rewards as data

A reward function decides what an experiment measures, and in most codebases it is the hardest thing to see: arithmetic spread through a training loop, changed by editing code, compared across papers by reading diffs. Here a reward is configuration — a named, serialisable composition of independent components — and this chapter explains the model. It also settles where a reward comes from at all, which is the environment contract's job.

Where the reward comes from: the two tiers

The package runs against any Gymnasium environment, at one of two tiers. This is the canonical statement of the contract; skyfall-crl conformance checks exactly this.

Tier 1 is any gymnasium.Env. It supplies its own reward from step(), and the core gives it everything that does not require seeing inside a step: regime scheduling, tracing, the training harness, the metrics, export. A Tier-1 environment is not a lesser citizen — it is the reason the environment layer names no substrate.

Tier 2 additionally describes what happens, with three capabilities the harness probes for:

1. A per-step record: info["trace"] carries an EpisodeStep built from a StepContext — the read-only slice of what the step did (state before and after, incidents raised, verifications, free-form extras).
2. A regime report: info["regime"] with the active label and a boundary flag, so the trace segments without the harness supplying labels.
3. Provenance: describe() returns what the run was configured with.

The step record is what unlocks this chapter: the composable reward system scores a StepContext, so only a Tier-2 environment can hand scoring over to it. A Tier-1 environment keeps its own reward and everything else still works.

The model

Four pieces, each doing one job:

flowchart LR
    SC["StepContext<br/>the step, described"] --> C1["component"] & C2["component"] & C3["component"]
    C1 & C2 & C3 --> E["engine:<br/>clip · weight · gates · sum"]
    E --> R["RewardResult<br/>total + per-component<br/>contributions"]
    R --> T["trace row"]

A component turns one aspect of a step into one number. It declares a name, a cadence (STEP fires every step; EPISODE_END fires once, on the episode's final step), a default clip and weight, and reads — the StepContext fields it consumes, which is how the engine can explain a zero instead of producing one silently. Seven ship with the package, all reading operational records; a new one is a small class registered by name.

A specification is the reward as data: which components, at what weights and clips, with what gates. It round-trips through YAML —

name: balance
components:
  - {type: upright, weight: 1.0, clip: [0.0, 1.0]}
  - {type: on_time_delivery, weight: 0.5}

— so the reward an experiment used is something its configuration states, not something its code implies. A gates: rule can zero one component's contribution unless another's is positive, for terms that should pay out only alongside real progress.

The engine runs the composition each step: skip components whose cadence has not come due, clip each raw value, apply the weight, apply the gates, sum. It returns the total and the per-component contributions, and it warns when a specification reads nothing the environment provides — the alternative being a reward that is silently zero forever.

The trace row carries the total and the breakdown (reward_components, and each component's sub-terms in reward_detail), which is what lets the evaluation layer attribute behaviour to terms months after the run.

The breakdown, concretely

The block below defines a minimal component, composes it with a second, and scores one step. It is executed when the documentation is built, and the output shown is the output produced.

from skyfall_crl.rewards import (
    Cadence, CompositionEngine, RewardComponent, RewardSignal, RewardSpec, StepContext, register,
)

class Upright(RewardComponent):
    name = "upright"
    cadence = Cadence.STEP
    reads = ("extras",)
    clip = (0.0, 1.0)
    default_weight = 1.0

    def compute(self, ctx: StepContext) -> RewardSignal:
        return self._signal(float(ctx.extras.get("upright", 0.0)))

class Effort(RewardComponent):
    name = "effort"
    cadence = Cadence.STEP
    reads = ("extras",)
    clip = (-1.0, 0.0)
    default_weight = 0.2

    def compute(self, ctx: StepContext) -> RewardSignal:
        return self._signal(-float(ctx.extras.get("effort", 0.0)))

register("upright", Upright, overwrite=True)
register("effort", Effort, overwrite=True)

spec = RewardSpec.from_dict({
    "name": "balance",
    "components": [
        {"type": "upright", "weight": 1.0},
        {"type": "effort", "weight": 0.2},
    ],
})
engine = CompositionEngine(spec)
result = engine.compute_step(StepContext(step=0, extras={"upright": 0.9, "effort": 0.25}))
print(f"total         = {result.total:.3f}")
for component, contribution in result.contributions.items():
    print(f"{component:<13} = {contribution:+.3f}")

total         = 0.850
upright       = +0.900
effort        = -0.050

Every number is attributable: the total is the sum of the contributions, and each contribution is one component's clipped value times its weight. That property holds for every step of every run, because it is how the engine is built rather than something a run opts into.

There is no default reward, deliberately

Handing the reward system to an environment means naming a specification. Omitting the reward: section entirely is the Tier-1 path — the environment scores itself — but asking for a composed reward without saying which is an error, never a fallback. The shipped specifications all describe MORPHEUS's operational model; defaulting to one against an environment that reports something else would score a structural zero on every step, silently, for the whole run. A mistake that loud on day one is cheaper than one that quiet for a month.

Weights, clips and provenance

A component's clip and weight are defaults, and a specification may override both per use — the same component can count for more in one experiment than another without being rewritten. Because the maximum contribution of every clipped, weighted term is known from the specification alone, the theoretical ceiling of a reward is derivable rather than asserted — which is what anchors the performance gap. And because the specification is data, it travels: a run's trace, an exported policy's manifest and an experiment document all carry it, so "what was this trained on" has an answer that does not involve reading a diff.