Rewards — API reference¶
Every public symbol in skyfall_crl.rewards, generated from the source.
Generated page — built from the source docstrings, so this file is blank when read on GitHub. Run
mkdocs serveto read it locally, or read the docstrings in the module itself.
The component contract¶
base ¶
The reward-component contract: one interface every reward signal implements.
A reward is a weighted composition of independent signals, each historically a bespoke computer with its own call shape and its own firing cadence. This module gives them a single contract so a declarative spec can compose any set of them.
This module imports no environment. The shapes a MORPHEUS step is described in live in
skyfall_crl.rewards.views, and the components that read them in
skyfall_crl.rewards.components; importing this module pulls in neither.
It does not follow that the contract is free of MORPHEUS. Six of StepContext's
fields describe MORPHEUS's operational model and two are annotated with adapter classes --
the split moved the definitions out, not the fields. An environment that describes its work
differently supplies its own record through StepContext.extras, and a component
written for it reads that.
Three concepts:
StepContext-- the read-only op-log slice for one environment step (the input; each component reads only the subset of fields it needs).RewardSignal-- a component's self-describing scalar output: the rawvalueplus itsname,cliprange, and defaultweight.RewardComponent-- the interface: aCadence, aresetfor per-episode state, andcomputereturning aRewardSignal.
Components are deliberately source-agnostic: a component reads whatever fields of
StepContext it needs and is indifferent to whether those fields were
projected from a live world or replayed from a recorded trace.
Cadence ¶
Bases: Enum
When a component contributes reward.
STEP
Fires every environment step -- e.g. the failure-event penalty and the
per-step chaos-ticket shaping terms.
EPISODE_END
Fires once, on the final step of an episode, from an accumulated
ledger/throughput snapshot -- e.g. cost variance and throughput
utilisation, which are only meaningful over a whole episode window.
RewardSignal
dataclass
¶
One component's contribution for a single evaluation.
Self-describing, so the composition engine and the per-component logger need only the signal -- not a back-reference to the component that produced it.
Attributes¶
value:
The raw scalar the component computed, before any spec-level weighting.
name:
Stable identifier of the producing component (e.g. "failure").
clip:
The (low, high) range the raw value is defined within. This is
declared metadata: a component is responsible for keeping its own value
in range; recording the bound here lets a spec and the upper-bound
toolkit reason about a component's reachable range.
weight:
The component's default weight, used when a spec does not override it.
Weights are research variables and need not sum to 1.
detail:
Optional per-sub-term breakdown for a component that internally sums
several terms (e.g. the chaos-ticket component). None for simple
scalar components; consumed by per-component logging.
StepContext
dataclass
¶
The read-only op-log slice for a single environment step.
This is the union of everything the reward components read; each component consumes only the subset it needs. Every operational field is optional so a caller -- a unit test, the environment adapter, or an offline trace replayer -- can construct exactly the slice a given component requires.
failure_types, tickets_before, tickets_after, verifications,
ledger and throughput describe MORPHEUS's operational model and are
populated by its adapter. Everything else is substrate-neutral, and
extras is the seam any other environment uses.
Attributes¶
step:
Zero-based step index within the episode.
failure_types:
Failure/chaos type of each incident ticket generated at this step
(None for an unmapped type). Drives the failure-event signal.
tickets_before, tickets_after:
Incident-ticket states immediately before and after the step's action.
Drive progress / solve / lifecycle terms.
verifications:
Verifier results for the tickets touched this step.
action, action_valid, action_result:
The agent's action, whether it passed validation, and its execution
result. action_valid=False drives the invalid-action penalty.
ledger:
Financial snapshot (e.g. planned_cost, actual_cost). Read by the
episode-end ledger signal.
throughput:
Throughput snapshot (e.g. units_processed, capacity). Read by the
episode-end throughput signal.
episode:
Episode-level scalars (e.g. steps_taken, max_steps). Read by
episode-end components that score the whole rollout.
prev_state, curr_state:
Optional raw world-state snapshots bracketing the step.
regime_id:
The active configuration id z_t. Present for logging and metric
segmentation only -- a component must not condition its reward on it, as
it is withheld from the policy observation by design.
observed_at:
When this step happened. Supplied, a signal that compares against a deadline
scores identically however long afterwards it is computed; omitted, such a signal
reads the clock, which is the behaviour the reference has.
is_last_step:
Whether this is the final step of the episode; gates EPISODE_END
components.
extras:
Anything else the environment wants a component to be able to read. The
fields above describe MORPHEUS's operational model; an environment that
describes its work differently puts its own record here, and a component
written for it reads it back. Keeping this open is what lets the reward
system score an environment the built-in components know nothing about.
RewardComponentProtocol ¶
Bases: Protocol
Structural view of a reward component, for duck-typed registries.
Any object exposing name, cadence, reset, and compute
satisfies this -- inheriting RewardComponent is the usual route but
not required.
RewardComponent ¶
Bases: ABC
Base class for a single reward signal.
A component declares its identity (name), when it fires
(cadence), the range of its raw value (clip), and its
default_weight. Subclasses implement compute; a component
that tracks progress or history across steps also overrides reset.
Set the four class attributes and build the output with _signal,
which stamps the declared name / clip / weight onto the value so
those live in exactly one place.
reset ¶
Clear any per-episode state.
Called once at the start of each episode, before the first
compute. The default is a no-op -- stateless components need not
override it.
compute
abstractmethod
¶
Return this component's RewardSignal for ctx.
Implementations read only the StepContext fields they need and
return _signal with the computed raw value.
Domain views a component may read¶
views ¶
Domain shapes a MORPHEUS step is described in.
These are adapter-side, not part of the reward system's contract. The core interface --
RewardComponent, RewardSignal, and the generic fields of StepContext -- knows nothing
about them, so an environment with no incident tickets and no verifier still has a reward system.
An environment that describes its work differently supplies its own shapes and reads them back
through StepContext.extras; see docs/plugins.md.
CheckResultView
dataclass
¶
One verifier check's result within a VerificationView.
VerificationView
dataclass
¶
A ticket's verifier result for one step.
progress is derived (passed_checks / total_checks) so it can never
drift from the underlying counts.
TicketView
dataclass
¶
The incident-ticket fields the reward components read.
Built-in components — chaos¶
chaos ¶
Chaos-ticket reward component: the eight-term per-step incident-solving signal.
Wraps the full ticket-solving reward as one stateful RewardComponent.
The eight terms share episode state (per-ticket progress, per-ticket passing
checks, a churn counter), so they are computed together and the composite is
emitted as one signal, with the decomposition carried in the signal's detail
for per-component logging.
ChaosBreakdown
dataclass
¶
Decomposition of one chaos-ticket step reward into its eight terms.
total is the three rewards minus the five penalties.
as_detail ¶
Flat {term: value} mapping including total (for signal detail).
ChaosTicketComponent ¶
Bases: RewardComponent
Eight-term chaos-ticket solving reward, wrapped as one signal.
Rewards verifier progress, ticket solves, and valid lifecycle transitions;
penalises open-ticket dwell time, missed deadlines, invalid actions, verifier
regressions, and churn (valid actions that make no progress). All ticket-
scoped terms are priority-weighted. Stateful within an episode — call
reset at episode start.
The emitted RewardSignal carries value = total and the eight
terms in detail. The last computed decomposition is also kept on
last_breakdown.
reads
class-attribute
instance-attribute
¶
reads = (
"tickets_before",
"tickets_after",
"verifications",
"action_valid",
"step",
"observed_at",
)
valid_lifecycle_transition ¶
Whether before → after is a permitted ticket-lifecycle transition.
Built-in components — failure¶
failure ¶
Failure-event reward component: r_f(t) = -Σ s(τ) (paper Appendix A).
FailureComponent ¶
Bases: RewardComponent
Severity-weighted incident-ticket penalty.
Sums the severity of every failure type generated at the step and negates it,
so the signal is 0 on clean steps and increasingly negative as more (and
more severe) failures occur. Contributes 0 under the upper-bound
assumption of no tickets generated. Reads StepContext.failure_types.
Built-in components — financial¶
financial ¶
Financial-ledger reward component: r_l = clip(1 - actual/planned, -1, 1).
LedgerComponent ¶
Bases: RewardComponent
Cost-variance reward relative to plan.
On budget scores 0; under budget positive (up to +1); overruns
negative (down to -1). Neutral (0) when there is no planned cost to
measure against, so an inactive window is not spuriously rewarded or
penalised. Reads planned_cost / actual_cost from
StepContext.ledger.
Built-in components — financial_profit¶
financial_profit ¶
Profit-based financial reward component.
An episode-level financial signal distinct from the paper-Appendix ledger term
(1 - actual/planned): normalised profit from realised revenue vs. costs. Used
by the experiment reward.
ProfitComponent ¶
Bases: RewardComponent
Normalised profit reward: clip((total_revenue - total_costs) / scale, 0, 1).
Reads total_revenue (completed order amounts) and total_costs (delivered
shipment costs) from StepContext.ledger. Clipped to [0, 1] — the
agent is rewarded for profit, not double-penalised for a loss.
Built-in components — severity¶
severity ¶
Failure-type severity weights (paper Appendix A).
The failure-event reward penalises each incident ticket by the severity of the failure type that produced it. The eleven types fall into four bands; concrete values within each band are ordered by operational impact so the table is reproducible.
SEVERITY_WEIGHTS
module-attribute
¶
SEVERITY_WEIGHTS: dict[str, float] = {
"permission_denied": 1.0,
"dependency_failure": 1.0,
"data_corruption": 0.8,
"missing_data": 0.7,
"invalid_state": 0.6,
"rate_limit": 0.4,
"format_change": 0.35,
"partial_data": 0.3,
"stale_data": 0.2,
"duplicate_data": 0.15,
"timing_issue": 0.1,
}
severity_weight ¶
Return the severity s(τ) for a failure type.
Lookup is case-insensitive; unmapped or missing types fall back to
DEFAULT_SEVERITY.
Built-in components — step_efficiency¶
step_efficiency ¶
Step-efficiency (brevity) reward component.
Rewards solving an episode in fewer steps, incentivising concise action sequences. Episode-level; complements the per-step churn penalty (which punishes local stagnation) by rewarding overall brevity. Used by the experiment reward.
StepEfficiencyComponent ¶
Bases: RewardComponent
Brevity reward: clip((max_steps - steps_taken) / max_steps, 0, 1).
1.0 for a near-instant solve, approaching 0 at the step budget, never
negative. Reads steps_taken / max_steps from StepContext.episode.
Built-in components — throughput¶
throughput ¶
Resource-throughput reward component: r_p = clip(units/capacity, 0, 1).
ThroughputComponent ¶
Bases: RewardComponent
Throughput-utilisation reward.
Rewards keeping resources productive — fulfilled orders and processed jobs —
relative to the configuration's capacity ceiling. Returns 0 when there is
no capacity to utilise. Reads units_processed / capacity from
StepContext.throughput.
Built-in components — verification¶
verification ¶
Verification-progress reward component (per-step pass-fraction shaping).
VerificationProgressComponent ¶
Bases: RewardComponent
Rewards per-step improvement in verifier pass-fraction.
The pass-fraction over a step is Σ passed_checks / Σ total_checks across
the step's verifier results; the reward is the non-negative improvement over
the previous step (max(0, frac_t - frac_{t-1})), so only forward progress
is credited. Stateful within an episode — call reset at episode start.
This is the per-step ("shaped") form of the verification-progress reward. The episode-terminal variants (whole-episode "solved" credit, or scoring only the final step) belong to the composition/environment layer, not to this per-step component.
Specs¶
spec ¶
Declarative reward specification — compose N components into one reward.
A RewardSpec lists the components to combine, each with optional weight
and clip overrides and constructor params, plus how their contributions
aggregate. It is serialisable (RewardSpec.to_dict / from_dict),
so a reward is a piece of configuration rather than hardcoded arithmetic.
Aggregation ¶
Bases: str, Enum
How component contributions combine into the step reward.
ComponentSpec ¶
Bases: BaseModel
One component's placement in a RewardSpec.
type names a component in the registry. weight and clip override
the component's own defaults when set (None keeps the default). params
are constructor keyword arguments (e.g. the chaos component's betas).
GateRule ¶
Bases: BaseModel
Conditionally zero a component's contribution for the step.
component's weighted contribution is set to 0 unless at least one
component named in unless_any_positive has a positive contribution that
step — e.g. gating a brevity reward so it only pays out alongside real task
progress.
RewardSpec ¶
Bases: BaseModel
A declarative, serialisable N-term reward.
Round-trips through to_dict / from_dict (thin wrappers over
the pydantic model_dump / model_validate); file and CLI loading are
layered on top by the configuration surface.
Named specs¶
specs ¶
Named reward specifications reconciling the MORPHEUS reward stacks.
Each of the historically-divergent rewards is available as a named, selectable spec — so an experiment picks a reward by name instead of forking code.
The composition engine¶
compose ¶
Composition engine — evaluate a RewardSpec into a step reward.
Reproduces the composite-reward semantics generically: per-step (STEP)
components fire every step; episode-level (EPISODE_END) components fire once,
on the final step; each fired signal is clipped, weighted, and aggregated.
RewardResult
dataclass
¶
The composed reward for one step.
Attributes¶
total:
The aggregated scalar reward for the step.
contributions:
Each fired component's weighted contribution, keyed by component name.
signals:
The raw RewardSignals that fired this step — each carries the
component's unweighted value, clip, weight, and any detail breakdown.
CompositionEngine ¶
Runs a RewardSpec's components and aggregates their signals.
Stateful: it owns one instance of each component (some track per-episode
state), so use one engine per environment and call reset at each
episode start. Vectorised environments use one engine each.
declared_reads ¶
The StepContext fields this spec's components say they read.
unmet_reads ¶
Declared fields this context does not provide.
Useful before a run: if this returns everything the spec declared, the spec and the environment are describing different worlds and every reward will be zero.
compute_step ¶
Compose one step's reward from the spec's components.
compute_episode ¶
Reset, then compose a reward for each step context in order.
Selecting a spec¶
config ¶
Reward configuration surface — turn a config value into a RewardSpec.
An experiment (or a world's declared reward) names its reward as a registered
spec name, an inline dict, or a path to a YAML/JSON file.
resolve_reward_spec normalises any of those into a
RewardSpec the composition engine can run.
RewardConfig ¶
Bases: BaseModel
The reward slice of an experiment configuration.
spec selects the reward: a registered name, an inline spec dict, or a path
to a YAML/JSON file. resolve turns it into a RewardSpec.
Required, deliberately. The shipped specs describe MORPHEUS, so defaulting would hand
an environment that reports something else a reward it can never earn -- silently, and
for the whole run. Naming the reward is cheap; discovering months later that it was always
zero is not. DEFAULT_SPEC remains available for callers
that do want the MORPHEUS default, and the environment adapter uses it.
resolve_reward_spec ¶
Normalise a reward-config value into a RewardSpec.
Accepts a registered spec name ("paper"), an inline dict, a
path to a YAML/JSON spec file, or an existing RewardSpec
(returned as-is).
The component registry¶
registry ¶
Reward-component registry — resolve a spec's type name to a class.
The registry is the seam custom reward sources plug into: a component registered
here by name can be named in any RewardSpec
without touching the composition engine. See docs/plugins.md and
examples/custom_reward_plugin.py for the plugin contract and a worked example.
The built-ins are resolved on first use, not on import. They describe MORPHEUS's operational model, and importing the composition engine should not drag that in: the engine composes whatever it is given, and an environment with no incident tickets should not pay for a severity table it will never read.
Custom registrations are a layer over the built-ins: registering a name shadows any built-in of that name, and removing your registration reveals the built-in again. A built-in with nothing shadowing it cannot be removed -- it is the package's, not the caller's.
Naming a component costs nothing: len, iteration, in, keys() and available
resolve no imports. Asking for values necessarily does, so items(), values(),
dict(...) and == import every built-in. That is inherent, and pinned by test.
The trace record¶
trace ¶
Per-step reward trace record + JSONL writer/reader.
The reward system logs each step's decomposition as an EpisodeStep,
serialised one per line to a traces.jsonl file. The six-metric evaluation
library reads these records back, so the schema carries exactly what those
metrics need: the per-step reward, each component's contribution, the regime
label z_t (for analysis only), and ticket/verifier snapshots.
EpisodeStep ¶
Bases: BaseModel
One time-step of a logged episode.
regime_id (the configuration id z_t) is present for analysis only — it
is never placed in the policy observation. reward_components holds each
component's weighted contribution; reward_detail holds any per-component
sub-term breakdown (e.g. the chaos component's eight terms).
Unrecognised keys are kept, not dropped. A TraceWriter merges run
metadata (run_id / seed / task / policy_id, or anything else a
caller supplies) into every row, and a metric that aggregates across seeds or
policies needs exactly those keys to survive the read. Reach them through
run_meta. The cost is that a misspelled field is accepted rather than
rejected — worth paying for a format whose extra keys are the point.
reward_components
class-attribute
instance-attribute
¶
reward_detail
class-attribute
instance-attribute
¶
ticket_snapshot
class-attribute
instance-attribute
¶
verifier_snapshot
class-attribute
instance-attribute
¶
run_meta
property
¶
The run metadata carried alongside this row, or {} if it carries none.
TraceWriter ¶
Appends EpisodeStep rows to a JSONL trace file.
Optional meta (e.g. run_id / seed / task / policy_id) is
merged into every row, matching the sandbox traces.jsonl layout, and reads
back on the EpisodeStep. Usable as a context manager.
A row's own keys win over meta, so a trace read and written again keeps the
provenance it was read with rather than being restamped.
step_from_result ¶
Build an EpisodeStep from a step context and its composed reward.
action overrides ctx.action when given (the executed tool call).
read_trace_jsonl ¶
Read a JSONL trace into EpisodeStep records, run metadata included.
Whatever a TraceWriter merged into each row is preserved and reachable
through EpisodeStep.run_meta.