Skip to content

Score a run you already have

The evaluation layer reads trace files and nothing else, so a run produced by any system scores identically — the whole contract is one JSON object per step with step and reward, and every further field sharpens what can be measured.

Write rows (from anything — this block fakes four), then point eval at them:

import json

rows = [
    {"step": 0, "reward": 0.2, "regime_id": "before", "run_id": "mine"},
    {"step": 1, "reward": 0.4, "regime_id": "before", "run_id": "mine"},
    {"step": 2, "reward": 0.1, "regime_id": "after", "run_id": "mine"},
    {"step": 3, "reward": 0.5, "regime_id": "after", "run_id": "mine"},
]
with open("yours.jsonl", "w") as handle:
    for row in rows:
        handle.write(json.dumps(row) + "\n")
print("wrote yours.jsonl:", len(rows), "rows")
skyfall-crl eval --traces yours.jsonl
1 trace(s), 2 configuration(s)

  Metric                    segment
  ---------------------------------
  Per-configuration reward      0.3
  Adaptation speed                2
  Forgetting                     --
  ...

What sharpened what: regime_id gave the run two configuration intervals, which is what made adaptation measurable at all; forgetting stays -- because no configuration recurs (regime_origin on a revisit is what pairs encounters); adding reward_components per row would light up the per-component readings, and identity fields (run_id, seed, algorithm_id) are what let a directory of files group into runs and families. --traces takes a directory tree and is repeatable; a ceiling comes from --spec or --upper-bound when you want the gap.