Benchmark metrics¶

The problem. A single average reward cannot say whether a policy is any good at continual learning. Two policies can earn the same total while behaving completely differently around a configuration change: one recovers in three steps and holds, the other never recovers but happened to start high. What distinguishes them is behaviour at the boundaries — which requires segmenting a run by configuration and measuring each segment against a reference.

The shape here. The six-metric protocol plus its supplementary signals, computed from recorded traces and nothing else — no environment, no model, no deployment, no ML stack — so a run is measurable long after the machine that produced it is gone, and an environment that has nothing to do with MORPHEUS is measured identically, because everything needed is on the row. The metrics chapter defines every number formally, with worked examples; this page is the library's practical surface.

What you supply: traces, and — for the ceiling-anchored readings — a reward specification or an explicit bound. The default anchor needs neither.

Scoring and comparing¶

from skyfall_crl.eval import compute_all_metrics, load_traces

runs = load_traces("runs/")                 # every run under the tree, identity from the rows
table = compute_all_metrics(runs)           # the protocol, segment-anchored

From the shell, skyfall-crl eval --traces runs/ prints the same table, and skyfall-crl aggregate --traces runs/ --baseline steady compares families — grouping by algorithm_id, scoring each family's seeds, reporting mean ± spread per family with the adaptation advantage against a baseline. Every flag is in the CLI reference; L5 reads a real cross-family table with discipline.

Three properties of the library worth knowing rather than discovering:

Absence is a value. A configuration seen once has no forgetting; a shift at a run's end has no adaptation window; an unmeasured plasticity column has no reading. All return nothing — never 0.0 — and aggregates drop them rather than averaging them in.
Anomalies are reported, not absorbed. A family whose seeds produced identical runs, a family pooling more than one task, a truncated run — each becomes a note under the table. Read the flags before trusting a spread.
Segmentation follows the label, and forgetting follows the origin. Every metric reports per label, so a revisit stays its own row; forgetting alone pairs encounters through what the label repeats, so both names report one number. An environment that labels its own steps supplies the origin as info["regime"]["origin"]; without it, a revisit reads as an unrelated configuration and forgetting is simply unavailable.

Anchors, and the ceiling¶

Adaptation speed and recovery time compare a running mean to a reference, and the choice — --anchor segment (default: the interval's own peak and settled tail) or --anchor ceiling (a theoretical bound) — changes the question being asked. The ceiling itself is derived, never guessed: from a reward specification's own clips and weights (--spec), numerically for a specification with unbounded components, or from the data (--data-relative, which moves with the policy measured and says so). Passing a specification also names its episode-end components, which adds the policy-attributable settled-reward column and lets forgetting exclude an episode-end payout that would otherwise dominate it.

Reading traces produced elsewhere¶

The trace format is the whole contract: one JSON object per step, identity on every row. A run split across files is rejoined through its identity; files with no identity at all stay separate runs. The minimal conforming row is {"step": 0, "reward": 0.0} — everything else sharpens what can be measured, and Score a run you already have is the recipe.

Limits¶

The library measures what a trace contains: no labels means no boundary-anchored metrics; no recorded effective rank means no plasticity reading. A gated specification's analytic ceiling is valid but generous — a gate only ever zeroes a contribution. The solve-rate column is opt-in because the component that marks solved work belongs to whichever environment produced the trace. And two definitional subtleties are recorded precisely rather than smoothed over: the performance gap and the settled reward measure slightly different tails and different occurrences (Fidelity has both), and numbers produced by the earlier research implementation of these metrics are not comparable — forgetting's sign is inverted, among other differences (Fidelity).

API: every public symbol, with signatures — Benchmark metrics — API reference.