Imagine you're running a restaurant kitchen where the head count changes every night. Some cooks show up, some don't. You can't fit everyone in the kitchen at once, so you keep a roster on the wall (disk), call in whoever tonight's menu needs (VRAM working set), and fire anyone who hasn't been requested in months. The cooks who stay aren't the ones who look busy — they're the ones other cooks actually pass dishes to. That's the core mechanism of mini-AGI: a mixture-of-experts language model where the experts live on disk, get paged into GPU memory by demand, grow by recombination when capacity is short, and get pruned when nothing routes to them. The committed claim: you can train a language model from scratch on a single 8 GB VRAM consumer GPU, with continual learning from a streaming byte sequence, without catastrophic forgetting, by letting the architecture assemble and disassemble itself. This is not a fine-tuning story or a LoRA adapter story. The model reads bytes, takes a gradient step per chunk, and the same forward pass serves both training and generation. No frozen base, no separate inference regime. Architecturally, the model belongs to the recurrent mixture-of-experts family with adaptive compute depth. Two dense prelude blocks feed into a single recurrent block applied up to 24 times per character. Each application independently selects top-8 experts from a shared pool, so one character can touch far more than 8 experts across its full depth. A PonderNet-style halting head lets easy characters exit early (as few as 4 applications) while hard characters use the full stack. Rotary positional encodings carry no learned parameters, meaning the context window extends naturally with continued training. The byte-level tokenization (all 256 byte values) eliminates vocabulary engineering entirely. The paging system is where the engineering gets interesting. Every expert is a file on disk holding weights and Adam optimizer moments. A two-tier cache (RAM, then VRAM) manages residency. Before each chunk, the model scores demand based on hidden states from the previous chunk — not raw embeddings — so the routing decision has contextual information. Hysteresis prevents thrashing: a candidate must beat a resident by a margin, and newcomers get a dwell period. Critically, Adam moments travel with the expert file, not with the VRAM slot. This prevents the insidious bug where a freshly-loaded expert inherits a stranger's momentum history. The growth and pruning logic is the most novel piece. New experts are created by recombination — whole hidden units spliced from several existing experts — because clones add nothing the router doesn't already have, and random initialization computes nothing worth routing to. Growth is gated by five simultaneous brakes: disk/VRAM room, utilization of existing additions, survival of the previous cohort, trial-period headroom, and train/held-out gap. Pruning keys on address frequency, not gate magnitude, and the authors call this their single most useful finding: gate magnitude is anti-predictive of usefulness. The busiest experts have the smallest gates; high-gate experts that nothing routes to look alive on a gate test but are functionally dead. The validation is honest about its limits. At 243M characters of training, the model produces grammatically correct, on-topic continuations but repeats itself. The authors explicitly label this a 'small toy-level model' and do not claim frontier capabilities. No benchmark comparisons to other models are provided — the claim is about the training regime and architecture properties (continual learning without forgetting, self-assembling capacity, consumer hardware), not about output quality. Weights are not yet published; the first full pass over the corpus is weeks away. What makes this worth watching is not what the model produces today but what the design constraints force. If continual learning from a single data stream without catastrophic forgetting actually works at this scale — and the architecture genuinely self-regulates its capacity — the implications for personal, sovereign models are real. The gap between 'toy that doesn't forget' and 'useful model that doesn't forget' is enormous, but this project is at least running the right experiment to find out where it breaks.
