Continual reinforcement learning¶
This page is the one piece of field background the rest of the documentation assumes. It defines the setting this package serves, distinguishes the two experimental shapes continual RL takes, and places the package — and MORPHEUS, the platform it was built alongside — within them. Everything after it is about the package itself.
The setting¶
Standard reinforcement learning makes a quiet assumption: the world holds still. The environment is a fixed Markov decision process, the agent trains until its policy converges, and evaluation asks how good that final policy is. Most benchmarks are built to match — episodic, quickly reset, and stationary by design.
Continual reinforcement learning drops the assumption. The term goes back to Ring's 1994 thesis, and the modern formulation is blunt: a continual learning agent is one for which learning never ends, because the world it acts in never stops changing (Abel et al., 2023). The case for taking that seriously is the big world hypothesis: for most problems worth having, the world is orders of magnitude larger than the agent, so no fixed policy can be adequate everywhere and the agent must keep learning with the capacity it has (Javed & Sutton, 2024). Deployed systems make the same case empirically — demand drifts, failure modes come and go, infrastructure degrades — and a policy frozen at deployment decays with them.
Learning continually raises problems that stationary training never poses:
- Catastrophic forgetting. Adapting to new conditions can overwrite what worked under the old ones, so when the old conditions return the policy has lost them (Kirkpatrick et al., 2017).
- Loss of plasticity. The mirror image: trained long enough, standard deep networks gradually lose the ability to fit anything new at all (Dohare et al., 2024).
- Detecting change. The agent is typically not told when conditions changed. Noticing the shift is part of the task, and an agent handed the label has had the hard part done for it.
- Measuring any of this. A single return says nothing about adaptation or forgetting, so the field measures behaviour around change — which is what the metrics chapter is about.
For a survey of the field, see Khetarpal et al. (2022).
Two experimental shapes¶
Continual-RL experiments arrange non-stationarity in one of two ways, and the difference decides what infrastructure they need.
| Task sequence | Parameter drift in a persistent world | |
|---|---|---|
| The world | a sequence of distinct environments, each its own MDP | one long-lived environment whose parameters shift while the agent acts |
| Reset | cheap and frequent; each task resets normally | destructive — resetting rebuilds the world and discards its accumulated state |
| Episodes | genuine terminal states | often none; the world simply continues |
| A "change" | the next task begins | the same world's conditions move underneath the agent |
| Example | Continual World (Wołczyk et al., 2021) | MORPHEUS (Seshadri et al., 2026) |
The two shapes are both legitimate, and the package spans both — the DiscoveryWorld tutorial runs a task-sequence setting where a change rebuilds the world while the policy carries across. Its home setting, though, is the second: a persistent world, drifting conditions, no reset worth having. That setting is under-served by episodic tooling, and it is where the vocabulary the next chapters define — regimes, configuration intervals, windows that are not episodes — comes from.
MORPHEUS, and where this package came from¶
MORPHEUS is a persistent enterprise simulation platform for continual-RL research, built by Skyfall AI: live business worlds — logistics, warehousing, order flows — that run continuously, inject structured failures, and shift their configuration on a schedule while an agent remediates the incidents the simulation raises. Policies are scored by verifiable rewards computed from the world's own operational record. It is introduced in MORPHEUS: A Persistent Enterprise Benchmark for Continual Reinforcement Learning in the Big World (Seshadri, Kumar, Chhabra, Tonpekar, Ganapathi Subramanian, Suleman, Pasupalak & Pasupalak, 2026).
skyfall-crl — this package — is the continual-RL infrastructure that was built alongside that
benchmark, generalised so that none of its layers requires MORPHEUS: composable verifiable
rewards, regime scheduling, a substrate-neutral environment layer, a config-driven training
harness, the six-metric evaluation protocol, and policy export. One adapter,
skyfall_crl.env.morpheus, speaks to a live MORPHEUS deployment; every other layer works with any
Gymnasium environment. The names differ for a plain reason: the
platform is MORPHEUS, the company is Skyfall AI, and the package carries the company's name
because it outgrew the single platform.
References¶
- Abel, Barreto, Van Roy, Precup, van Hasselt & Singh (2023). A Definition of Continual Reinforcement Learning. NeurIPS 2023. arXiv:2307.11046
- Dohare, Hernandez-Garcia, Lan, Rahman, Mahmood & Sutton (2024). Loss of plasticity in deep continual learning. Nature 632, 768–774. nature.com/articles/s41586-024-07711-7
- Javed & Sutton (2024). The Big World Hypothesis and its Ramifications for Artificial Intelligence. RLC 2024, Finding the Frame workshop. openreview.net/forum?id=Sv7DazuCn8
- Khetarpal, Riemer, Rish & Precup (2022). Towards Continual Reinforcement Learning: A Review and Perspectives. JAIR. arXiv:2012.13490
- Kirkpatrick et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS 114(13). arXiv:1612.00796
- Ring (1994). Continual Learning in Reinforcement Environments. PhD thesis, University of Texas at Austin.
- Seshadri, Kumar, Chhabra, Tonpekar, Ganapathi Subramanian, Suleman, Pasupalak & Pasupalak (2026). MORPHEUS: A Persistent Enterprise Benchmark for Continual Reinforcement Learning in the Big World. openreview.net/pdf?id=31P1VAfLkJ
- Wołczyk, Zając, Pascanu, Kuciński & Miłoś (2021). Continual World: A Robotic Benchmark for Continual Reinforcement Learning. NeurIPS 2021. arXiv:2105.10919