Comparing algorithms

One run is not a result. A comparison worth reporting is several algorithm families, over several seeds, under identical conditions — and then the spread, not just the mean. This rung writes that comparison as a single document, runs the whole matrix, and reads the cross-family table it produces, including the part where the honest answer is no measurable difference.

The environment, unchanged

The L2 environment, exactly as built there:

# file: windy.py
"""CartPole whose gravity shifts on a fixed period, with a penalty for dropping the pole."""
import gymnasium as gym


class Windy(gym.Wrapper):
    def __init__(self, period: int = 200, calm: float = 9.8, windy: float = 18.0):
        super().__init__(gym.make("CartPole-v1"))
        self._period, self._calm, self._windy = int(period), float(calm), float(windy)
        self._count = 0

    def _apply(self) -> None:
        phase = (self._count // self._period) % 3
        self.unwrapped.gravity = self._windy if phase == 1 else self._calm

    def reset(self, *, seed=None, options=None):
        self._apply()
        return super().reset(seed=seed, options=options)

    def step(self, action):
        self._apply()
        self._count += 1
        observation, reward, terminated, truncated, info = super().step(action)
        if terminated:
            reward -= 5.0
        return observation, reward, terminated, truncated, info


def make(**kwargs):
    return Windy(**kwargs)

The matrix, as one document

# file: sweep.yaml
base:
  env:       {id: "windy:make", kwargs: {period: 200}}
  algorithm: {name: discrete_hill_climbing, params: {seed: "${seed}"}}
  rollout:   {window_steps: 100, total_steps: 600, auto_reset: true}
  regime:
    provider: scheduled
    params: {schedule: {regimes: [
      {regime_id: calm, duration_steps: 200},
      {regime_id: windy, duration_steps: 200},
      {regime_id: calm_again, alias_of: calm, duration_steps: 200}]}}
  run: {seed: "${seed}"}
families:
  steady: {}
  bold:   {algorithm: {params: {seed: "${seed}", noise: 0.8}}}
seeds: [1, 2, 3]
output_dir: runs

base: is what every cell shares. Each entry under families: is one arm — here the same hill climber at two mutation scales, which is a legitimate comparison: a family is an arm of the experiment, not necessarily a different algorithm. The family's name becomes each run's algorithm_id, the label the table groups on.

${seed} is load-bearing. It is substituted wherever it appears, and it appears inside the algorithm's parameters because that is what actually varies between seeds here — the policy's random initialisation and mutations. run.seed alone seeds only the environment; a sweep whose seed reaches nothing else produces identical runs per family, a spread of exactly zero, and a table that reads as impossibly certain. The aggregator flags that pattern when it sees it.

See it, then run it

skyfall-crl run --matrix sweep.yaml --dry-run

6 run(s):
  steady-seed1  algorithm=discrete_hill_climbing  seed=1  task=--  steps=600  trace=runs/steady-seed1/traces.jsonl
  steady-seed2  algorithm=discrete_hill_climbing  seed=2  task=--  steps=600  trace=runs/steady-seed2/traces.jsonl
  ...

A matrix can be worth hours of compute, so expansion is separate from execution: the dry run prints every cell and runs nothing. This one is six cells of a fast toy, so:

skyfall-crl run --matrix sweep.yaml --plugin windy.py
skyfall-crl aggregate --traces runs --baseline steady

Cross-family comparison

6 trace(s) grouped by algorithm_id: bold (3 seeds), steady (3 seeds)
Advantage measured against: steady

  Metric                                      bold                  steady
  ------------------------------------------------------------------------
  Per-configuration reward     0.844444 ± 0.123697     0.838889 ± 0.132899
  Adaptation speed                           1 ± 0                   1 ± 0
  Forgetting                0.00833333 ± 0.0381881  0.00833333 ± 0.0381881
  Recovery time                  3.16667 ± 3.75278       2.66667 ± 2.88675
  Stability (variance)         0.743579 ± 0.577507      0.767728 ± 0.61743
  Settled reward              0.861111 ± 0.0636469         0.875 ± 0.11024
  Zero-shot reward             0.916667 ± 0.144338     0.916667 ± 0.144338
  Return (discounted)            84.2504 ± 20.3281       83.5312 ± 20.6739
  Return (undiscounted)          506.667 ± 74.2181       503.333 ± 79.7392
  Invalid-action rate                        0 ± 0                   0 ± 0
  Adaptation advantage                           0                      --

Read it honestly

One column per family; every cell is a mean over that family's seeds ± the sample spread across them. The last row is the relative adaptation advantage against the named baseline — positive would mean bold adapts faster than steady; both adapt in one step here, so it is 0.

And the honest verdict of this table is: these two families are indistinguishable at three seeds. Every reward-shaped row overlaps by far more than it differs — 0.844 ± 0.124 against 0.839 ± 0.133 is not a difference, it is noise wearing a decimal point — and recovery's spreads (±3.8, ±2.9) swallow its means outright. That is not a failed experiment; it is what most honest comparisons look like at small scale, and the discipline is to say so rather than read a winner off the second decimal. Sharper questions need more seeds, a longer run, or a metric with more resolution — which is precisely where the full experiment goes.