Rewards¶

The problem. A reward for a long-lived operational environment is not one number with one formula. It is several signals — incidents raised, money moved, work completed, checks passed — that fire at different moments, live on different scales, and need reweighting per experiment. Written as code, every change to the objective is a change to the trainer.

The shape here. A reward is a list of weighted, clipped components in a serialisable specification. Components are named and registered; a specification picks, weights and clips them; the engine runs it each step and reports the total and each component's contribution. Changing the objective is editing data; adding a signal is registering one class. Rewards as data teaches the model with a live example — this page is the layer's practical surface.

What you implement: a RewardComponent — a name, a cadence, a clip range, a default_weight, reads, and a compute that scores a StepContext. Extending is the recipe; skyfall-crl conformance --reward is the check.

The built-in specifications¶

Four named specifications ship, corresponding to the reward implementations that coexist in the MORPHEUS research code, reconciled here into one composable system:

Spec	Components	Corresponds to
`paper`	`failure` (0.5) + `ledger` (0.25) + `throughput` (0.25)	the reward as the paper's appendix states it: incident severity, cost variance, and units processed
`experiment`	`verification_progress` + `financial_profit` + `step_efficiency` (⅓ each, gated)	the reward the paper's experiments actually trained on
`eval8`	`chaos_tickets` (one eight-term component)	the ticket-solving reward the reference online evaluation stepped with
`verification_progress`	`verification_progress`	the standalone per-step verifier pass-fraction signal

Each is numerically verified against the reference code it reconciles — the method and volumes are in What is verified; which of the three formulations to treat as canonical is a research question, recorded rather than resolved, in Fidelity.

There is no default, deliberately¶

Every shipped specification reads MORPHEUS's operational fields, so applying one to an environment that reports something else scores zero on every step, silently, forever. Asking for a composed reward therefore means naming one — RewardConfig requires its spec — and the engine warns once when the specification you chose can read nothing from the steps it is given. The one exception is MorpheusEnv, which is a MORPHEUS environment: name nothing and it adopts the world's own declared reward, falling back to paper.

Selecting a reward¶

By registered name, a file, or an inline document — all through one resolver:

from skyfall_crl.rewards import RewardConfig, resolve_reward_spec

resolve_reward_spec("paper")                    # a built-in, by name
resolve_reward_spec("configs/my_reward.yaml")   # a YAML/JSON file
resolve_reward_spec({"name": "custom", "components": [{"type": "failure", "weight": 1.0}]})

RewardConfig(spec="eval8").resolve()            # the experiment document's reward section

The four built-ins also ship as editable YAML templates under skyfall_crl/rewards/spec_files/, and the full document schema is in the configuration reference.

Weights, clips, gates¶

A component declares defaults; a specification overrides either per use ({"type": "failure", "weight": 0.5, "clip": [-1, 0]}), so the same signal can count for more in one experiment than another without being rewritten. Contributions are clipped, weighted, gated, summed — and every step's reward is emitted with its per-component breakdown onto the trace, which is what lets a result be attributed term by term months later. The declared clips are also what make the reward's theoretical ceiling derivable.

Limits¶

One aggregation ships (weighted_sum); weights are static within a run; a gate has one shape (zero a contribution unless another is positive). A component reads the fields the step record carries — new inputs come from the environment's instrumentation, not from configuration.

API: every public symbol, with signatures — Rewards — API reference.