Conformance¶
The problem. Four things in this package are meant to be written by somebody else: an environment, a reward component, an algorithm backend, and a configuration provider. Each contract is documented in prose, and prose is not checkable — so the only way to discover an implementation was subtly wrong has been to wire it into a run and interpret whatever came out. Worse, the most damaging mistake announces nothing: an environment that puts the active configuration where the policy can read it trains perfectly well and produces adaptation numbers that mean nothing, and a leaked label looks exactly like a well-populated observation.
The shape here. Each contract as something you run — against your own code, with nothing from this repository:
Naming an environment and an algorithm together also drives a short real run — a configuration document in, a trace out, the metrics computed from it — because an implementation can satisfy every method and still be untrainable. The run uses the backend a base install ships, so none of this needs an ML stack.
What you implement: nothing — this is the checker for the things the other pages have you
implement. From Python, check_environment, check_reward_component, check_algorithm and
check_provider return the same reports the command renders.
What is checked¶
For an environment, tier 1 checks the Gymnasium contract as this package relies on it; tier 2 adds the three Tier-2 capabilities and the label-leak check the whole benchmark rests on:
ok declares an observation space and an action space
ok reset() returns an observation inside its space
ok step() returns a well-formed transition
ok releases what it holds when closed
ok the same seed starts the same way
ok describe() reports what the run was configured with
ok reports the active configuration in info
ok emits an EpisodeStep for every step
ok the configuration is not visible in the observation
A reward component is driven over a fully-populated synthetic step: the signal must be finite and
inside its declared clip, reset() must be safe to repeat, and every name in reads must be a
real step-record field — the typo that would otherwise make the engine's
reads-nothing warning misfire. An algorithm's policy must act inside the environment's space and
its update must return finite metrics. A provider must answer for any step and answer the same
step the same way — the property every segmentation rests on.
Two severities¶
A required failure means it will not work. A recommended one means it works and gives something up — a stochastic environment is not reproducible under a seed; an algorithm that does not checkpoint cannot be exported. The exit code reflects required failures alone, because a tool that fails implementations which are fine is a tool people learn to ignore.
Limits¶
Conformance is necessary, not sufficient: it checks the contracts this package can express, and the DiscoveryWorld integration found its real costs in the parts no protocol describes. The reward check drives a synthetic record, so a component that only works against one live substrate passes here and still needs that substrate to be useful. And the label-leak check inspects the observation the environment returns — a label an observation wrapper adds afterwards is outside its sight.
API: every public symbol, with signatures — Conformance — API reference.