A full experiment¶
The capstone. Everything below happens the way a real study does: a question is stated, a matrix of runs answers it, the answer is read with discipline, one cell is reproduced exactly, and the winning policy leaves the experiment as an artifact — exported, served, and scored again in inference. Every command and number on this page is executed and verified when the documentation is built.
flowchart LR
Q["question"] --> M["sweep matrix<br/>families × seeds"]
M --> T["traces<br/>one per cell"]
T --> A["aggregate<br/>cross-family table"]
T --> E["eval<br/>per-run deep dive"]
M --> B["export<br/>policy bundle"]
B --> S["serve<br/>inference trace"]
S --> EThe question¶
Does mutation scale matter when the world shifts underneath the policy? The
L4 comparison put steady (noise 0.3) against bold (0.8) at three
seeds and honestly could not tell them apart. This experiment widens both axes: a third family —
cautious, noise 0.05, mutations twenty times smaller — and five seeds instead of three.
The environment is L2's shifting world, byte for byte:
# file: windy.py
"""CartPole whose gravity shifts on a fixed period, with a penalty for dropping the pole."""
import gymnasium as gym
class Windy(gym.Wrapper):
def __init__(self, period: int = 200, calm: float = 9.8, windy: float = 18.0):
super().__init__(gym.make("CartPole-v1"))
self._period, self._calm, self._windy = int(period), float(calm), float(windy)
self._count = 0
def _apply(self) -> None:
phase = (self._count // self._period) % 3
self.unwrapped.gravity = self._windy if phase == 1 else self._calm
def reset(self, *, seed=None, options=None):
self._apply()
return super().reset(seed=seed, options=options)
def step(self, action):
self._apply()
self._count += 1
observation, reward, terminated, truncated, info = super().step(action)
if terminated:
reward -= 5.0
return observation, reward, terminated, truncated, info
def make(**kwargs):
return Windy(**kwargs)
The matrix¶
# file: sweep.yaml
base:
env: {id: "windy:make", kwargs: {period: 200}}
algorithm: {name: discrete_hill_climbing, params: {seed: "${seed}"}}
rollout: {window_steps: 100, total_steps: 600, auto_reset: true}
regime:
provider: scheduled
params: {schedule: {regimes: [
{regime_id: calm, duration_steps: 200},
{regime_id: windy, duration_steps: 200},
{regime_id: calm_again, alias_of: calm, duration_steps: 200}]}}
run: {seed: "${seed}"}
families:
steady: {}
bold: {algorithm: {params: {seed: "${seed}", noise: 0.8}}}
cautious: {algorithm: {params: {seed: "${seed}", noise: 0.05}}}
seeds: [1, 2, 3, 4, 5]
output_dir: runs
15 run(s):
steady-seed1 algorithm=discrete_hill_climbing seed=1 task=-- steps=600 trace=runs/steady-seed1/traces.jsonl
steady-seed2 algorithm=discrete_hill_climbing seed=2 task=-- steps=600 trace=runs/steady-seed2/traces.jsonl
...
Fifteen cells, each an ordinary run with its own trace. Run and aggregate:
skyfall-crl run --matrix sweep.yaml --plugin windy.py
skyfall-crl aggregate --traces runs --baseline steady
Cross-family comparison
15 trace(s) grouped by algorithm_id: bold (5 seeds), cautious (5 seeds), steady (5 seeds)
Advantage measured against: steady
Metric bold cautious steady
-------------------------------------------------------------------------------------
Per-configuration reward 0.786667 ± 0.134578 0.705 ± 0.242069 0.776667 ± 0.139866
Adaptation speed 1 ± 0 2.6 ± 3.57771 1 ± 0
Forgetting -0.09 ± 0.137614 -0.065 ± 0.188414 -0.1 ± 0.153093
Recovery time 2.3 ± 2.90689 4.9 ± 3.57771 4 ± 4.47214
Stability (variance) 1.00494 ± 0.614763 1.34158 ± 1.07531 1.04858 ± 0.638103
Settled reward 0.808333 ± 0.133723 0.725 ± 0.261207 0.791667 ± 0.164042
Zero-shot reward 0.85 ± 0.136931 0.7 ± 0.273861 0.9 ± 0.136931
Return (discounted) 72.5597 ± 21.6945 68.0075 ± 25.6544 71.688 ± 21.9299
Return (undiscounted) 472 ± 80.7465 423 ± 145.241 466 ± 83.9196
The answer, read with discipline¶
Three findings, in decreasing order of confidence:
- The continual-learning metric separates where the return does not.
steadyandboldadapt in one step, every seed — 1 ± 0.cautiousreports 2.6 ± 3.6: some of its seeds also snap back, and some take seconds of simulated time to re-find their footing after the gravity shift, because mutations twenty times smaller sometimes cannot climb out of what the shift broke. The plain returns (472 vs 423 vs 466, spreads ~±100) overlap far too much to say any of that. This is the protocol's reason to exist: behaviour around change is visible where aggregate return is noise. cautioustrends worse on every column — reward, recovery, stability, zero-shot — but trends are all they are; each mean sits inside its neighbours' spreads.steadyversusboldremains a tie at five seeds, as it was at three. An honest table says so.
The cautious family's ±0.24 on reward is itself worth a look, and one command drills into a
single cell:
1 trace(s), 3 configuration(s)
Metric segment
----------------------------------
Per-configuration reward 0.466667
Adaptation speed 1
Forgetting 0
Recovery time 8
Stability (variance) 2.39405
Performance gap --
Zero-shot reward 0.5
Regret --
Effective rank --
Settled reward 0.458333
Return (discounted) 49.381
Return (undiscounted) 280
Invalid-action rate 0
Seed 5 earned 0.47 per step against its family's 0.705 mean: this seed's policy never coped at all, in any interval. The family's wide spread is not diffuse noise but two sub-populations — seeds that adapted and seeds that never did — which a mean ± spread can hint at and only a per-cell look can show. That is what the per-run tables are for.
Reproduce a cell exactly¶
A result that cannot be re-run is an anecdote. Every cell here is deterministic given its document — the seed reaches the environment and the policy — so reproducing one is running it again:
# file: best.yaml
env: {id: "windy:make", kwargs: {period: 200}}
algorithm:
name: discrete_hill_climbing
params: {seed: 1, noise: 0.8}
policy:
id: skyfall_crl.train.backends.hill_climbing:linear_menu_policy
kwargs: {n_actions: 2, seed: 1}
rollout: {window_steps: 100, total_steps: 600, auto_reset: true}
regime:
provider: scheduled
params: {schedule: {regimes: [
{regime_id: calm, duration_steps: 200},
{regime_id: windy, duration_steps: 200},
{regime_id: calm_again, alias_of: calm, duration_steps: 200}]}}
run: {seed: 1, trace_path: best.jsonl}
This is the bold, seed-1 cell written out as a standalone document — with one addition: the
policy is named rather than left to the algorithm's internal default, because the export
below must record how to rebuild it.
from pathlib import Path
from skyfall_crl.train import ExperimentConfig, run_experiment
config = ExperimentConfig.from_yaml_file("best.yaml")
config.run.trace_path = "again.jsonl"
run_experiment(config)
first = Path("best.jsonl").read_text().splitlines()
second = Path("again.jsonl").read_text().splitlines()
print(f"rows: {len(first)} and {len(second)}, identical: {first == second}")
Six hundred rows, byte-identical. On this toy nothing was left unseeded; with the ML backends two draws are not seeded by default and the scaling section below sets them explicitly.
The policy leaves the experiment¶
--export above wrote bundle/: a manifest recording the environment, the observation and
action spaces, the policy's constructor and weights format, and the run's provenance — beside the
checkpoint itself. Serving it back needs nothing from this page but the environment file:
skyfall-crl serve --bundle bundle --steps 200 --auto-reset --seed 7 --traces served.jsonl --plugin windy.py
1 trace(s), 1 configuration(s)
Metric segment
----------------------------------
Per-configuration reward 0.95
...
The served policy earns 0.95 per step over 200 inference steps — one early fall, then flawless —
with nothing updating, the same trace format, the same scorer. Three details are worth noticing.
--seed 7 makes the served run repeatable; inference against an unseeded environment is
legitimately different every time, and a scored inference run should not be. --auto-reset is
needed here for the same reason it was in training: CartPole genuinely ends episodes, and serving must
say so explicitly because against a persistent world an automatic reset would be destructive.
Finally, the served run reports one configuration, and the reason is worth getting right:
the bundle recorded the training run's schedule, and serve replays it by default so a served
trace segments exactly as a training trace does — but 200 inference steps sit entirely inside
the schedule's first 200-step calm interval, so one segment is what there is. Serve longer and
the shifts return; pass --stationary to opt out of the recorded schedule deliberately.
Scaling this to the ML backends¶
Everything above is the full workflow at toy scale. Scaling it to the shipped language-model backends changes the contents of the documents and none of their shape:
base:
env: {id: "yourpkg.envs:make"}
rollout: {window_steps: 512, total_steps: 100000}
run: {seed: "${seed}"}
families:
ppo:
algorithm:
name: ppo
params: {seed: "${seed}", replay_seed: "${seed}"}
policy:
id: skyfall_crl.train.backends.policy:language_policy
kwargs: {model: ./sft-checkpoint}
ppo_ewc:
algorithm:
name: ppo_ewc
params: {seed: "${seed}", replay_seed: "${seed}", fisher_seed: "${seed}"}
policy:
id: skyfall_crl.train.backends.policy:language_policy
kwargs: {model: ./sft-checkpoint}
seeds: [1, 2, 3, 4, 5]
output_dir: runs
What changes: the backends (ppo, ppo_ewc, ppo_lcm) need the [train] extra, a GPU worth
using, and a policy built from a real checkpoint; replay_seed and fisher_seed must be stated
because the ported algorithms deliberately do not seed those draws themselves; windows are
hundreds of steps and totals are days, not seconds. What does not change: the document's shape,
--dry-run before committing the compute, the aggregate table and its discipline, the per-cell
deep dive, the reproduction check, and the export. That is the point of the ladder — the
workflow you just ran is the workflow.
What this page did not claim¶
The finding is real but the world is a toy: nothing here was trained to convergence, and a hill climber on CartPole earns no publication. What the capstone demonstrates is the experiment machinery — question to matrix to disciplined answer to reproducible, servable artifact — so that when the environment is one that matters, the machinery is already yours.