Skip to content

Instrumenting to Tier 2

The windy world changes, but its reward is still nearly binary — upright pays 1, a fall pays once — so adaptation and recovery had almost nothing to resolve. This rung instruments the same environment to Tier 2: it will describe every step, score itself through a composed reward specification, label its own trace, and report its provenance. The measurements sharpen immediately, and the duplication L2 warned about disappears.

The reward, as a file

# file: balance.yaml
name: balance
components:
  - {type: upright, weight: 1.0}

One component, reward-as-data style. The same file will score the run and derive its ceiling — one source of truth for what the experiment optimises.

The instrumented environment

# file: windy2.py
"""The windy environment, instrumented to Tier 2: it labels and describes every step."""
import gymnasium as gym

from skyfall_crl.rewards import (
    Cadence,
    CompositionEngine,
    RewardComponent,
    RewardSignal,
    RewardSpec,
    StepContext,
    register,
    step_from_result,
)

REGIMES = ("calm", "windy", "calm_again")


class Upright(RewardComponent):
    """How vertically the pole is held: 1 upright, 0 at the failure angle."""

    name = "upright"
    cadence = Cadence.STEP
    reads = ("extras",)
    clip = (0.0, 1.0)
    default_weight = 1.0

    def compute(self, ctx: StepContext) -> RewardSignal:
        return self._signal(float(ctx.extras.get("upright", 0.0)))


register("upright", Upright, overwrite=True)

SPEC = RewardSpec.from_yaml_file("balance.yaml")


class WindyDescribed(gym.Wrapper):
    def __init__(self, period: int = 200, calm: float = 9.8, windy: float = 18.0):
        super().__init__(gym.make("CartPole-v1"))
        self._period, self._calm, self._windy = int(period), float(calm), float(windy)
        self._count = 0
        self._engine = CompositionEngine(SPEC)

    def _phase(self) -> int:
        return (self._count // self._period) % len(REGIMES)

    def _apply(self) -> None:
        self.unwrapped.gravity = self._windy if self._phase() == 1 else self._calm

    def reset(self, *, seed=None, options=None):
        self._apply()
        self._engine.reset()
        return super().reset(seed=seed, options=options)

    def step(self, action):
        self._apply()
        regime = REGIMES[self._phase()]
        boundary = self._count % self._period == 0
        observation, _, terminated, truncated, info = self.env.step(action)
        upright = max(0.0, 1.0 - abs(float(observation[2])) / 0.2095)
        ctx = StepContext(
            step=self._count,
            regime_id=regime,
            regime_origin="calm" if regime == "calm_again" else None,
            extras={"upright": upright},
        )
        scored = self._engine.compute_step(ctx)
        info["regime"] = {"id": regime, "boundary": boundary}
        info["trace"] = step_from_result(ctx, scored)
        self._count += 1
        return observation, scored.total, terminated, truncated, info

    def describe(self):
        return {"environment": "windy-described", "period": self._period,
                "reward_spec": SPEC.name}


def make(**kwargs):
    return WindyDescribed(**kwargs)

Walk it against the Tier-2 contract — three additions, one per capability:

  1. The step record. Each step builds a StepContext — here just the pole's uprightness (1 vertical, 0 at the failure angle) in extras, the seam meant for exactly this — has the engine score it, and puts the resulting EpisodeStep in info["trace"]. The environment's reward is the composition's total, so what the trace explains and what the policy earned cannot disagree.
  2. The regime report. info["regime"] carries the label and a boundary flag, and the StepContext carries the label and its origin onto the trace. The environment now owns its own labels — which is why the configuration below has no regime: section at all. The L2 duplication is gone.
  3. Provenance. describe() says what this run was configured with; it travels into the run's record and, later, into an exported policy's manifest.

Run it

# file: experiment.yaml
env:       {id: "windy2:make", kwargs: {period: 200}}
algorithm: {name: discrete_hill_climbing}
rollout:   {window_steps: 100, total_steps: 600, auto_reset: true}
run: {name: described, seed: 0, trace_path: traces.jsonl}

skyfall-crl run --config experiment.yaml --plugin windy2.py
skyfall-crl eval --traces traces.jsonl --spec balance.yaml --plugin windy2.py

eval now takes the spec too — the same file — and derives the ceiling from it. The --plugin matters there as well: deriving a ceiling resolves the components the spec names.

1 trace(s), 3 configuration(s), reward spec 'balance'

  ceiling (analytic): 1 per step, 1 on the episode's last

  Metric                     analytic
  -----------------------------------
  Per-configuration reward   0.698133
  Adaptation speed                 57
  Forgetting                0.0885529
  Recovery time                     3
  Stability (variance)      0.0786697
  Performance gap            0.283771
  Zero-shot reward           0.692865
  Regret                           --
  Effective rank                   --
  Settled reward             0.716229
  Return (discounted)         80.4604
  Return (undiscounted)        418.88
  ...

What the instrumentation bought

Compare this against L2's table, same world, same algorithm:

  • Adaptation speed 1 → 57. Not a worse policy — a reward that can finally see. Uprightness is continuous, so the running mean climbs as the policy genuinely re-learns the doubled gravity, and crossing 90% of its eventual level takes 57 real steps. The 1 was the measurement saturating; the 57 is the measurement working.
  • Forgetting flipped sign: −0.175 → +0.0886. Under the graded reward, the revisit is slightly worse than the first encounter — genuine, mild forgetting the binary reward could not detect.
  • Performance gap 0.284, at last. The spec's one component clips to [0, 1] at weight 1, so the derived ceiling is 1 per step — and the settled state of 0.716 leaves a measured 0.284 on the table. No hand-supplied bound involved.
  • Every trace row now carries the breakdown. reward_components: {upright: ...} on each step — one component here, but the same rows attribute every future spec term by term.

Check it, then trust it

The contract has a checker, and an instrumented environment should pass it before anything is built on top:

from skyfall_crl.conformance import check_environment, render

import windy2

print(render([check_environment(windy2.make, tier=2)]))

environment (tier 2): make
--------------------------
  ok    declares an observation space and an action space
  ok    reset() returns an observation inside its space
  ok    step() returns a well-formed transition
  ok    releases what it holds when closed
  ok    the same seed starts the same way
  ok    describe() reports what the run was configured with
  ok    reports the active configuration in info
  ok    emits an EpisodeStep for every step
  ok    the configuration is not visible in the observation

The last line is the one the whole benchmark rests on: the label reaches the trace and never the observation. From the shell, the same check is skyfall-crl conformance --env windy2:make --tier 2 --plugin windy2.py. With a conforming Tier-2 environment in hand, one run stops being the interesting unit — comparing algorithms is next.