Conformance — API reference¶
Every public symbol in skyfall_crl.conformance, generated from the source.
Generated page — built from the source docstrings, so this file is blank when read on GitHub. Run
mkdocs serveto read it locally, or read the docstrings in the module itself.
What a run produces¶
report ¶
What a conformance run produces: a list of checks, and two ways to read it.
Two severities, because two different things can be wrong. A required check failing means the plugin will not work — the harness will raise, or the metrics will be meaningless. A recommended one failing means it works and gives something up: runs that cannot be reproduced, or a policy that cannot be exported. Reporting them the same way would either fail implementations that are legitimately fine (a stochastic environment is not reproducible under a seed, and an algorithm need not checkpoint) or bury a real problem among advice. The exit code keys off required alone.
No pytest. This runs on a base install, which has none: a check is a function returning None
when it passes and a sentence when it does not, and an exception is caught and reported rather than
ending the run. That also means one broken check cannot hide the twelve after it.
Severity ¶
Check
dataclass
¶
Report
dataclass
¶
Every check run against one subject.
Recorder
dataclass
¶
Accumulates checks, turning a raised exception into a failure rather than a crash.
render ¶
The reports as a table a person reads, or JSON a script does.
Environments¶
environment ¶
Does this environment satisfy the contract the harness drives it through?
Two tiers, because the environment contract has two. Tier 1 is any gymnasium.Env: it supplies
its own reward, and the core gives it configuration scheduling, tracing, metrics, a harness and
export. Tier 2 adds a per-step record, and gets the composable reward system with it. A Tier-1
environment is not a lesser one; it is the whole point of the environment layer naming no substrate.
The check that matters most is the last one. A continual-learning benchmark asks the agent to notice that its world has changed; an environment that puts the configuration's name where the policy can read it has answered the question on the agent's behalf, and every adaptation number it produces afterwards is meaningless. Nothing else in the package can catch that, because a leaked label looks exactly like a well-populated observation.
contains ¶
Whether space accepts value, measured without the caller's warning filters.
Gymnasium warns when it has to cast -- a list where a Box is declared, say -- and a caller
running with warnings as errors would otherwise see a cast reported as a containment failure.
What is being asked here is whether the space accepts the value, and it does.
check_environment ¶
check_environment(
factory: Callable[[], Any],
*,
tier: int = 1,
steps: int = 12,
subject: str | None = None,
) -> Report
Run the environment contract against whatever factory builds.
Reward components¶
reward ¶
Does this reward component satisfy the contract the composition engine drives it through?
The awkward part of checking a reward component is that it needs something to read. A MORPHEUS
component wants incident tickets and verifier results; one written for a foreign environment reads
ctx.extras; and neither can be asked to supply a context in order to be checked. So the checks
drive a synthetic StepContext with every one of its eighteen fields populated — plausible
tickets, a verification, a ledger, a throughput snapshot, an action and its result, and an extras
mapping. A component reading any of them finds something, and one reading nothing is reported as
reading nothing, which is a finding rather than a pass.
The other thing worth checking is reads. It names the context fields a component looks at, and the
composition engine uses it to warn when a whole spec engages with nothing in its environment. A typo
there does not break the component; it breaks the warning, silently — which is exactly the kind of
defect nothing else would find.
CONTEXT_FIELDS
module-attribute
¶
sample_context ¶
A step context with every field populated, so any component finds something to read.
busy=False gives the same shape with nothing happening in it -- no incidents, no
verification progress, no money moved. The pair is what lets a component be asked whether its
value depends on its input at all; varying only the step number would leave most components
reading identical fields and looking like constants when they are not.
check_reward_component ¶
Run the reward-component contract against component.
Algorithm backends¶
algorithm ¶
Does this algorithm backend satisfy the contract the harness drives it through?
Four methods, two of which are optional in practice and checked as such. policy and update are
what a run cannot proceed without; on_regime_shift and save_checkpoint describe things a backend
may reasonably not do — a method that neither consolidates on a shift nor persists anything is a
legitimate algorithm that simply gives up continual-learning behaviour and exportability.
Algorithm is a runtime-checkable Protocol, which is worth not relying on: isinstance against
one checks that the methods exist and nothing about what they do. So the checks here drive the
backend — a real window in, metrics out — rather than asking whether it looks right.
sample_window ¶
A window with structure in it: rewards that vary, and an episode that ends inside it.
check_algorithm ¶
check_algorithm(
factory: Callable[[], Any],
*,
env_factory: Callable[[], Any] | None = None,
subject: str | None = None,
) -> Report
Run the algorithm contract against whatever factory builds.
Configuration providers¶
regime ¶
Does this configuration provider satisfy the contract the harness drives it through?
One method. RegimeProvider is regime_id(step) -> str and nothing else. That is worth saying
because it has been specified otherwise: an earlier design gave it get_current_config,
is_boundary and on_shift, and a suite written from that would reject ConstantRegime — this
package's own default. Boundaries are derived by watching the id change, which works for a schedule
and for a detector that has no schedule to consult.
The load-bearing property is not the signature but the purity: every metric that segments a run by configuration assumes the same step always reports the same id. A provider that consulted a clock or a counter would make two scorings of one trace disagree, and nothing downstream would notice.
A real run¶
integration ¶
Does it actually train? The check the contracts cannot make on their own.
Everything else here asks whether a piece satisfies its interface. This drives a short real run through the shipped path -- configuration document in, trace out, metrics computed from the trace -- because an implementation can satisfy every method and still be untrainable: an environment whose episodes never end, a reward that never varies, an action space a policy cannot produce a member of. That class of defect is what the adoption demo actually found, and no isolated contract call would have shown it.
It needs no ML stack: the run uses discrete_hill_climbing, the backend a base install ships.
check_integration ¶
check_integration(
*,
env_id: str,
env_kwargs: dict[str, Any] | None = None,
algorithm: str = "discrete_hill_climbing",
algorithm_params: dict[str, Any] | None = None,
regime: dict[str, Any] | None = None,
subject: str | None = None,
) -> Report
Run a short experiment end to end and score it.