Benchmark metrics — API reference¶
Every public symbol in skyfall_crl.eval, generated from the source.
Generated page — built from the source docstrings, so this file is blank when read on GitHub. Run
mkdocs serveto read it locally, or read the docstrings in the module itself.
The protocol¶
metrics ¶
The six-metric continual-learning protocol, plus its supplementary signals.
What the protocol measures. A policy is not judged by one number here. It is judged by how it behaves around a configuration change: how much reward it earns under each configuration, how quickly it recovers after one changes, whether it loses what it knew when an earlier configuration returns, how steadily it performs, and how far short of the achievable ceiling it settles.
1. Per-configuration reward ``J_k`` -- mean reward over configuration *k*
2. Adaptation speed ``tau_adapt`` -- steps after a shift to reach ``alpha * R_ub``
3. Forgetting mean(first encounter) - mean(second encounter)
4. Recovery time ``tau_rec`` -- steps for the running mean to settle near ``R_ub``
5. Stability within-configuration reward variance
6. Performance gap ``R_ub`` minus the settled reward
Supplementary: relative adaptation advantage, zero-shot reward, discounted return, optional regret, and a plasticity reading.
Everything is anchored to a theoretical upper bound, not to an oracle. R_ub comes from the
reward specification itself -- see skyfall_crl.eval.upper_bound -- which is what lets these
numbers be compared across policies without running a reference agent.
An unavailable metric is None\ , never zero. A configuration that ran once has no
forgetting; a run whose policy never measured its effective rank has no plasticity reading. Zero is
a value a metric can legitimately take, so using it to mean absent would silently corrupt every
average computed downstream.
Segment on the configuration label, never on window boundaries. A rollout window is an
administrative slice of a continuing run; a configuration interval is what the protocol measures
over. The two are unrelated, and only regime_id marks the second.
This module reads recorded traces and nothing else -- no environment, no model, no deployment.
AdaptationWindow ¶
Bases: str, Enum
Which steps count towards adaptation speed and recovery time after a shift.
SEGMENT bounds the window to the configuration's own interval. This is what the protocol
specifies -- adaptation and recovery are measured within the configuration interval -- and it
is the default.
POST_SHIFT takes every step from the shift onward, whatever configuration it ran under, so
slow adaptation to one configuration can be credited to reward earned under the next one if
another shift arrives first. Kept because one published implementation does exactly that, and a
number computed against it is only comparable to another computed the same way.
Anchor ¶
Bases: str, Enum
What adaptation and recovery are measured against.
SEGMENT (the default) is data-relative: adaptation asks when the running mean reaches a
fraction of the segment's own peak running mean, and recovery asks when it settles near the
segment's own tail mean. It answers how quickly did this policy get to where it was going.
CEILING is the theoretical formulation: a fraction of the reward a perfect policy could
reach. It is the right question when that ceiling is attainable -- and the wrong one when it is
not, which is why the default moved.
The reason this is a choice at all. Anchored to a ceiling no policy approaches, both metrics stop discriminating: every family caps at the interval length and recovery never triggers. The implementation this was ported from hit exactly that -- four algorithms all reporting the same adaptation speed -- and concluded that the ceiling is the wrong reference under a reward whose realistic values sit orders of magnitude below it. A ceiling remains the better anchor where it is a real bound rather than a sum of maxima that never co-occur.
EpisodeTrace ¶
Bases: BaseModel
One policy's recorded run: the rows, the discount, and who produced them.
Built from a trace file rather than declared by hand -- see
skyfall_crl.eval.ingest. run_meta carries whatever the writer merged into the
rows (run_id, seed, policy_id, task, or anything else), which is how a
multi-seed sweep groups without a side-channel manifest.
Deliberately no configuration schedule. Every boundary this module needs is derivable
from the rows' own regime_id, so a trace is self-describing and skyfall_crl.eval
depends on no scheduler.
default_alpha ¶
The adapted-fraction that means something for this anchor.
cumulative_reward ¶
Discounted return over the whole trace.
Discounted by position in the trace, not by the step index the rows carry, so a trace assembled from a partial run still discounts from its own beginning.
undiscounted_reward ¶
Plain sum of every step's reward.
per_configuration_reward ¶
Mean reward over every step that ran under one configuration.
Averaged across all of that configuration's occurrences, so a configuration that recurs
contributes both visits. None when it never ran.
adaptation_speed ¶
adaptation_speed(
trace: EpisodeTrace,
shift_step: int,
target_regime: str,
upper_bound: float | None = None,
alpha: float | None = None,
window: AdaptationWindow = AdaptationWindow.SEGMENT,
anchor: Anchor = Anchor.SEGMENT,
) -> int | None
Steps after a shift for the running mean reward to reach the adapted level.
What counts as adapted depends on Anchor. Under SEGMENT it is a fraction of the
segment's own peak running mean — when did the policy reach most of where it was going.
Under CEILING it is a fraction of upper_bound — when did it reach a workable absolute
level — which is only a useful question when that level is attainable.
alpha defaults per anchor (default_alpha); upper_bound is required only under
CEILING and ignored otherwise.
The two anchors also disagree about "never". Under SEGMENT a run that never crosses
returns the interval length, because a policy is always at its own peak by the end of the
interval that defines it. Under CEILING it returns None: "never got there" and "took
exactly this long" are different findings about an absolute target.
forgetting ¶
forgetting(
trace: EpisodeTrace,
regime_id: str,
*,
exclude_episode_end_bonus: bool = False,
episode_end_components: Collection[str] = (),
) -> float | None
How much worse a configuration goes the second time it is seen.
mean(first occurrence) - mean(second), so positive means the policy got worse and
negative means it improved -- backward transfer. None unless the configuration occurs at
least twice, which is why a schedule intended to measure this has to bring one back.
exclude_episode_end_bonus matters more than it sounds. A configuration's second
encounter typically contains the episode's final step, where episode-end terms pay out. That
payout is not something the policy earned in those conditions, and left in it can dominate the
second mean — so forgetting ends up measuring the bonus rather than forgetting. Name the
episode-end components (or derive them from the reward spec) and those steps are dropped.
A revisit under a new label still counts. A schedule may return to a configuration and name that return something else; the row records what it repeats, and this compares the two encounters through that rather than through the label. Both labels therefore report the same number, which is right -- forgetting is a property of the conditions, not of what a visit is called. Without this a schedule built to measure forgetting would report none for either name.
recovery_time ¶
recovery_time(
trace: EpisodeTrace,
shift_step: int,
target_regime: str,
upper_bound: float | None = None,
epsilon: float | None = None,
window: AdaptationWindow = AdaptationWindow.SEGMENT,
anchor: Anchor = Anchor.SEGMENT,
epsilon_fraction: float = DEFAULT_EPSILON_FRACTION,
epsilon_abs_floor: float = DEFAULT_EPSILON_ABS_FLOOR,
) -> int | None
Steps after a shift for the running mean to settle.
Under Anchor.SEGMENT it settles towards the segment's own tail mean — its
asymptote — inside a band of max(epsilon_fraction × |asymptote|, epsilon_abs_floor). The
absolute floor is load-bearing rather than cosmetic: without it a settled state at or near zero
gives a zero-width band and the metric can never trigger, which is the common case for a reward
whose realistic values are small.
Under Anchor.CEILING it settles towards upper_bound within epsilon, defaulting
to half the bound. Under that anchor's defaults this asks the same question adaptation asks
— recovery wants a mean in [0.5·R, 1.5·R], adaptation one at or above 0.5·R, and below
1.5·R those coincide, which is why published tables show a recovery time of exactly 1.0
wherever adaptation was fast.
stability ¶
Reward variance within a configuration, per occurrence and pooled.
Returns {"per_segment": [...], "pooled": ...}. A single-step occurrence has no variance
and contributes None rather than 0.0, which would read as perfect steadiness.
tail_mean_reward ¶
tail_mean_reward(
trace: EpisodeTrace,
regime_id: str,
tail_fraction: float = DEFAULT_TAIL_FRACTION,
*,
exclude_episode_end_bonus: bool = False,
episode_end_components: Collection[str] = (),
) -> float | None
The settled reward itself, with nothing subtracted from it.
The mean over the final tail_fraction of each of the configuration's occurrences, averaged
across them. Where performance_gap reports the distance from a ceiling, this reports
the value — which is the more informative of the two whenever the ceiling is a sum of maxima
that never co-occur, because the gap then just restates the ceiling.
exclude_episode_end_bonus strips the steps that paid an episode-end term, leaving the part
of the settled state the policy is actually responsible for.
performance_gap ¶
performance_gap(
trace: EpisodeTrace,
regime_id: str,
upper_bound: float,
tail_fraction: float = DEFAULT_TAIL_FRACTION,
) -> float | None
How far the settled reward falls short of the ceiling.
upper_bound minus the mean reward over the final tail_fraction of the
configuration's first occurrence -- the settled state, after adaptation has had its
chance. None when the configuration never ran.
relative_adaptation_advantage ¶
relative_adaptation_advantage(
adapt_self: float | None, adapt_baseline: float | None
) -> float | None
How many steps faster than the baseline a policy adapted.
baseline - self, so positive means faster. Compares the same configuration across two
runs, which is why it takes two numbers rather than a trace. None if either is missing.
Takes floats rather than the integers a single trace produces, because the comparison that matters is between families, and a family's adaptation speed is a mean across its seeds.
zero_shot_reward ¶
Mean reward over the first delta steps after a shift -- before adaptation.
Truncated when fewer steps remain, so this is a mean over up to delta steps. None
when the shift is at or past the end of the trace.
regret ¶
Cumulative shortfall against a reference trajectory, when one exists.
None without one, rather than zero -- the protocol is deliberately oracle-free, so this
is available only when someone supplies a reference run.
plasticity ¶
Mean recorded effective rank, over the whole trace or one configuration.
A plasticity diagnostic: a policy whose effective rank falls without its reward recovering is losing the capacity to represent anything new.
This reads the column; it never computes one. Effective rank is measured from a policy's
activations at training time (skyfall_crl.train.plasticity) and recorded on the row,
which is what keeps this library free of an ML stack.
Capture is off by default, so an entirely unmeasured trace is normal and returns
None. Averaging the zeros would report a collapsed policy where there was simply no
measurement.
shift_boundaries ¶
Every configuration change in a trace, as (step, from, to).
Derived from the rows themselves, so a trace needs no schedule to be segmented -- and a detector that changes configuration without a schedule is measured the same way.
compute_all_metrics ¶
compute_all_metrics(
traces: Sequence[EpisodeTrace],
upper_bound: float | Mapping[str, float] | None = None,
*,
anchor: Anchor = Anchor.SEGMENT,
alpha: float | None = None,
epsilon: float | None = None,
tail_fraction: float = DEFAULT_TAIL_FRACTION,
zero_shot_window: int = DEFAULT_ZERO_SHOT_WINDOW,
window: AdaptationWindow = AdaptationWindow.SEGMENT,
solve_component: str | None = None,
episode_end_components: Collection[str] = (),
oracle_rewards: dict[str, Sequence[float]]
| None = None,
epsilon_fraction: float = DEFAULT_EPSILON_FRACTION,
) -> dict[str, Any]
The whole protocol over one or more seeds, per configuration and aggregated.
Aggregation is two-stage, as the protocol specifies: a metric is averaged across seeds within
a configuration, then across configurations. Missing values are dropped at both stages rather
than counted as zero, so a metric unavailable for one configuration does not drag the mean
down -- and a metric unavailable everywhere reports None.
anchor decides what adaptation and recovery are measured against, and the default is
data-relative — so a ceiling is optional. Pass one to get the performance gap, or to use
Anchor.CEILING; without one those columns simply report nothing.
episode_end_components names the terms that only pay at an episode's end — derive them from
the reward spec with
episode_end_components. Supplying them adds the
policy-attributable settled-state column and lets forgetting exclude a payout the policy did
not earn in those conditions.
upper_bound anchors the performance gap and, under CEILING, the other two; derive it
with skyfall_crl.eval.upper_bound rather than guessing, since every one of those
numbers moves with it. A mapping gives each configuration its own ceiling -- which is what
bounds_by_configuration produces, and what the interval
holding the episode's last step needs, since the episode-end terms can only land there. A
configuration the mapping does not cover raises rather than falling back to a plausible number.
solve_component names a reward component whose positive values mark a solved unit of
work, and is omitted when the traces do not carry one. It is a parameter rather than a
constant because the component's name belongs to whichever environment produced the trace.
oracle_rewards maps a run id to a reference trajectory, for the optional regret signal.
epsilon and epsilon_fraction are two different tolerances, one per anchor, and only
the one belonging to the active anchor has any effect. Under CEILING recovery settles
within epsilon of the bound, an absolute distance defaulting to half of it. Under
SEGMENT it settles within epsilon_fraction of the segment's own asymptote, a
proportion. Reproducing a published table sometimes means matching the fraction the analysis
used rather than the one its library defaulted to -- the two need not agree, and in the work
this was ported from they did not.
Reading recorded runs¶
ingest ¶
Reading recorded runs back, long after the run.
A trace file is a flat stream of rows, and a run's identity travels on every row rather than in a header — which is what lets a writer append to one file from several runs, and what lets a directory of files be read without a manifest describing what is in it. Reassembling runs from that stream is this module's whole job.
Nothing here needs the environment that produced the trace, or the policy, or a deployment. A sweep can be measured on a laptop from a directory someone sent you.
Two details decide the design, and both come from real recorded traces rather than from taste:
- Run identity is a named set of fields, not "everything the row carries." Real traces carry per-step values alongside the run metadata -- an observation hash, a log-probability, a value estimate. Grouping on everything would put every row in a run of its own.
- Run metadata is what does not vary within a run. Rather than hard-coding a list of what counts, a field is treated as belonging to the run when every row agrees on it. The per-step values fall out on their own, and a writer that records something nobody here anticipated still gets it carried through.
RUN_IDENTITY
module-attribute
¶
load_trace ¶
The runs recorded in one trace file.
Usually one, but a writer may append several to the same file, so this returns a list in the order each run's first row appears. Row order within a run is preserved exactly as written -- a trace is a sequence, and re-sorting it would quietly repair a file that is actually broken.
gamma overrides the discount for every run; without it, a run that recorded its own is
scored with that, and one that did not is scored at DEFAULT_GAMMA.
load_traces ¶
load_traces(
path: str | Path,
*,
gamma: float | None = None,
pattern: str = "*.jsonl",
) -> list[EpisodeTrace]
Every run under a path -- one file, or a directory tree of them.
This is how a multi-seed sweep is read: point it at the directory the runs were written to and
every seed comes back as its own trace, ready to hand to
compute_all_metrics together. No manifest is needed, because
each row already says which run it belongs to.
Files are read in sorted order, so the result is stable across machines. A run split across several files is reassembled, since identity travels on the row -- but runs carrying no identity are kept separate per file, because two anonymous files are far more likely to be two runs than one run written twice.
A file that is not valid JSONL raises, naming the file. A corrupt trace is worth stopping for; silently skipping it would report a sweep as complete when part of it was unreadable.
Comparing families of runs¶
aggregate ¶
Comparing families of runs.
Scoring a directory of traces pools them into one result. That answers "how did this go", and a continual-learning result is a different question: which of these algorithms handled the shifts better, and by how much, and is the difference bigger than the spread across seeds. This module answers that one.
Three things about it are decided by the protocol rather than by taste.
The aggregation order is per-seed first. The protocol computes a metric per configuration per seed, averages over configurations to get that seed's number, and then reports mean±standard deviation across seeds. Pooling every seed into one call gives a different answer whenever the seeds are unbalanced -- and gives no deviation at all, which is the number that says whether a gap between two families means anything.
One seed reports no deviation, not zero. A single run has no spread to measure. Reporting
0.0 would say the opposite of what is true: that the result is perfectly repeatable.
Nothing is silently excluded. A truncated run, a run that earned nothing, a run that never left its first configuration -- all of them stay in the table and are named in the notes. The work this was ported from carried a hardcoded list of ten specific runs to skip, which is how a table comes to describe a different experiment than its caption claims.
Grouping is by a field of the run metadata, algorithm_id by default. It is a parameter because
a recorded corpus may carry a field that looks like a family and is not: the published runs this
was checked against record the checkpoint there, so two different algorithms share a value and
one algorithm spans two. Where that happens, the file a run was read from is what separates them,
and SOURCE_FIELD is a groupable field like any other.
Aggregate
dataclass
¶
FamilyResult
dataclass
¶
Everything measured about one family.
per_seed
class-attribute
instance-attribute
¶
AggregateResult
dataclass
¶
The comparison.
metric ¶
One metric across every family, for rendering a row.
aggregate_runs ¶
aggregate_runs(
traces: Sequence[EpisodeTrace],
upper_bound: float | Mapping[str, float] | None = None,
*,
group_by: str = DEFAULT_GROUP_BY,
baseline: str | None = None,
min_steps: int | None = None,
**protocol: Any,
) -> AggregateResult
Compare families of runs, seed by seed.
protocol is passed straight to
compute_all_metrics -- anchor, alpha, epsilon, window and
the rest -- so a comparison is computed under exactly the parameters a single table would be.
group_by names the run-metadata field that identifies a family. baseline names the
family the adaptation advantage is measured against; without one, a family called ppo is
used if present and otherwise the first, since the advantage is defined against vanilla PPO.
min_steps excludes runs shorter than that, and says in the notes what it excluded. It is
off by default: a short run is usually worth seeing rather than hiding.
The upper bound¶
upper_bound ¶
The reward a perfect policy could earn, derived from the reward specification.
Three of the six metrics -- adaptation speed, recovery time, performance gap -- are measured against this ceiling rather than against a reference agent, which is what lets their numbers be compared across policies, seeds and environments. Get the ceiling wrong and all three move with it.
The ceiling is not one number. A term that only fires at the end of an episode cannot
contribute to any other step, so a step has a lower ceiling than the episode's last step does. That
distinction is the difference between a metric that discriminates and one that does not: the
implementation this was ported from defaulted to the episode ceiling for every step, roughly three
times the real one, and the performance gap consequently sat in [0.9722, 1.0000] for every
policy and configuration -- differences buried in the third decimal.
Nothing here needs a constant kept in step with the weights. Cadence
already records which terms can fire when, so re-weighting a specification re-derives its bound.
Three ways to get one, in the order to try them:
analytic_upper_bound-- the closed form, from the specification's own weights and clip ranges. Exact, needs no data, and raises rather than guessing when a component declares no upper bound.numeric_upper_bound-- run the reward over contexts you believe represent perfect play and take the best it scores. For specifications the closed form cannot bound.data_relative_upper_bound-- what a policy actually achieved. Useful for reading a run, and not a basis for comparing policies: it moves with the policy being measured.
Variant ¶
Bases: str, Enum
Where a bound came from, and therefore what it may be used for.
ANALYTIC is a property of the reward specification and is the same whoever is being
measured, so policies can be compared against it. DATA_RELATIVE is a property of a
particular run -- it moves with the policy that produced it, so two policies measured against
their own are not being compared with each other.
UnboundedReward ¶
Bases: ValueError
A specification whose closed-form ceiling is infinite.
Raised instead of substituting a plausible number, because substituting one is exactly the failure this module exists to correct.
UpperBound
dataclass
¶
The ceiling, on an ordinary step and on the episode's last one.
per_step is what a step that is not the episode's last can reach; final_step
additionally includes the terms that only fire at the end. They are equal when a specification
has no episode-end terms.
for_interval ¶
The ceiling on the mean reward over an interval of length steps.
This, not per_step, is what a configuration interval is measured against: the
metrics compare a running mean to the bound, and an interval containing the episode's last
step can reach slightly more than the others because the episode-end terms land in it.
Sixty steps of experiment, one of them the last, gives
(59 × 0.3333 + 1.0) / 60 = 0.3444 -- the figure the original derivation worked out by
hand.
analytic_upper_bound ¶
analytic_upper_bound(
spec: RewardSpec,
*,
registry: Mapping[str, Any] | None = None,
) -> UpperBound
The closed-form ceiling: every term at its maximum, split by when it can fire.
Read from the specification, so a re-weighted spec gets a re-derived bound with nothing to keep in step by hand.
Gates are ignored, and the result is therefore possibly loose. A gate can only zero a contribution, never raise it, so the ungated sum remains a valid ceiling.
Raises:
| Type | Description |
|---|---|
UnboundedReward
|
when a component declares no bound at the end its weight points to. |
episode_end_components ¶
episode_end_components(
spec: RewardSpec,
*,
registry: Mapping[str, Any] | None = None,
) -> frozenset[str]
The names of the spec's terms that only fire on the episode's last step.
Used to find that step in a recorded run: a row carrying a contribution from one of these is the final step, which is more reliable than assuming the last row of a file is.
numeric_upper_bound ¶
numeric_upper_bound(
spec: RewardSpec,
contexts: Iterable[StepContext],
*,
registry: Mapping[str, Any] | None = None,
quantile: float = 1.0,
) -> UpperBound
The best this reward scores over contexts you supply.
For specifications the closed form cannot bound. The contexts are the assumption: this returns a ceiling over what you gave it, so it is an upper bound only to the extent that they represent perfect play. Supply them as a trajectory, in order -- components carry state between steps, and a sequence of unrelated contexts scores differently from a run.
Contexts marked as the episode's last step set final_step; the rest set
per_step. With only one kind, both take the same value.
data_relative_upper_bound ¶
data_relative_upper_bound(
traces: Sequence[Any],
*,
spec: RewardSpec | None = None,
regime_id: str | None = None,
quantile: float = 1.0,
registry: Mapping[str, Any] | None = None,
) -> UpperBound
What was actually achieved, read off recorded runs.
Not a basis for comparing policies. It moves with whichever policy produced the traces, so two policies each measured against their own will both look similarly close to "the ceiling". Report it beside the analytic bound, which does not move, rather than instead of it.
It answers a different and genuinely useful question: how much of the reachable reward a run captured, when the analytic ceiling is so far above anything observed that the gap says nothing.
spec lets the episode's last step be identified from the per-component breakdown; without
it, the last row of each trace is assumed to be it.
bounds_by_configuration ¶
bounds_by_configuration(
bound: UpperBound,
traces: Any,
*,
spec: RewardSpec | None = None,
registry: Mapping[str, Any] | None = None,
) -> dict[str, float]
The ceiling each configuration in a run is actually measured against.
Takes one trace or a sequence of them; several are pooled rather than averaged, so a sweep whose seeds ran different lengths still gets the arithmetic right.
Ceilings differ only where the episode's last step falls, since that is the one step the episode-end terms can contribute to. For a specification without such terms every configuration gets the same number -- which is the correct answer, not a missing feature: a ceiling varies by configuration only when the reward's own limits do.
register_derivation ¶
register_derivation(
name: str,
derivation: Callable[..., UpperBound],
*,
overwrite: bool = False,
) -> None
Make a derivation available by name.
get_derivation ¶
The derivation registered under name.