Skip to content

Read the aggregate's flags

skyfall-crl aggregate reports anomalies as notes under the table rather than absorbing them — and the notes are the difference between a table and a result. The run below is deliberately degenerate to make it produce two:

import json

for seed in (1, 2):
    with open(f"arm-seed{seed}.jsonl", "w") as handle:
        for step in range(6):
            handle.write(json.dumps({
                "step": step, "reward": 1.0, "regime_id": "only",
                "run_id": f"arm-seed{seed}", "seed": seed, "algorithm_id": "arm",
            }) + "\n")
print("two seeds, identical rewards")
skyfall-crl aggregate --traces arm-seed1.jsonl --traces arm-seed2.jsonl
  arm: its 2 seeds produced identical runs, so any deviation reported across them is zero by construction. Check that the seed reaches the policy and the algorithm -- write ${seed} where it belongs in the sweep -- and not only the environment

  arm: arm-seed1 saw one configuration, so adaptation, recovery and forgetting have nothing to measure in it

What each note means, and what to do:

  • "identical runs" — the spread you are about to read as certainty is zero by construction: the seed varied nothing that mattered. Write ${seed} into the algorithm's and policy's parameters in the sweep document — the recipe — and rerun.
  • "saw one configuration" — the run was stationary, so the boundary-anchored metrics have nothing to measure; give the run a schedule, or read only the rows a stationary run supports.
  • "pools more than one task" — a family's mean is averaging across different tasks; group tasks apart (tasks: in the sweep gives each cell its label) or compare per task.
  • "shorter than" — a truncated run is being averaged with full ones; --min-steps N excludes short runs and says which were excluded — nothing is ever dropped silently.

A table whose notes you have read and resolved is a result; a table whose notes you scrolled past is a number.