Read the aggregate's flags¶
skyfall-crl aggregate reports anomalies as notes under the table rather than absorbing them —
and the notes are the difference between a table and a result. The run below is deliberately
degenerate to make it produce two:
import json
for seed in (1, 2):
with open(f"arm-seed{seed}.jsonl", "w") as handle:
for step in range(6):
handle.write(json.dumps({
"step": step, "reward": 1.0, "regime_id": "only",
"run_id": f"arm-seed{seed}", "seed": seed, "algorithm_id": "arm",
}) + "\n")
print("two seeds, identical rewards")
arm: its 2 seeds produced identical runs, so any deviation reported across them is zero by construction. Check that the seed reaches the policy and the algorithm -- write ${seed} where it belongs in the sweep -- and not only the environment
arm: arm-seed1 saw one configuration, so adaptation, recovery and forgetting have nothing to measure in it
What each note means, and what to do:
- "identical runs" — the spread you are about to read as certainty is zero by construction:
the seed varied nothing that mattered. Write
${seed}into the algorithm's and policy's parameters in the sweep document — the recipe — and rerun. - "saw one configuration" — the run was stationary, so the boundary-anchored metrics have nothing to measure; give the run a schedule, or read only the rows a stationary run supports.
- "pools more than one task" — a family's mean is averaging across different tasks; group
tasks apart (
tasks:in the sweep gives each cell its label) or compare per task. - "shorter than" — a truncated run is being averaged with full ones;
--min-steps Nexcludes short runs and says which were excluded — nothing is ever dropped silently.
A table whose notes you have read and resolved is a result; a table whose notes you scrolled past is a number.