
Imagine you're a home renovation show host. You can describe a gorgeous kitchen remodel in vivid detail — granite countertops, gas range, subway tile backsplash — and your audience nods along. But if you hand those descriptions to a contractor, half the walls don't line up with the plumbing. The description was plausible but not buildable. That's the state of LLM-generated cybersecurity exercises: fluent scenario text that falls apart the moment you try to actually stand it up on real infrastructure. ScenarioForge is the contractor's blueprint layer — it takes high-level prompts and produces not just descriptions but deployable, validated cyber-range environments. The committed claim: ScenarioForge turns natural-language exercise requirements into fully grounded, deployable, and runtime-validated cyber-range scenarios by constraining LLM generation with a resource vocabulary (294 Vulhub images, 85 custom challenge-resource generators) and validating every step against live infrastructure in the CORE emulator. This is not "LLMs write better exercise descriptions." It's "LLMs produce artifacts that actually compile, deploy, and pass runtime checks." The numbers are worth parsing carefully. The ScenarioForge Execution Dataset contains 226 scenarios, all of which passed construction, preview, sequencing, deployment, strict runtime validation, and live artifact checks — a 100% success rate across 867 individually verifiable challenge steps. In a matched 50-prompt benchmark comparing ScenarioForge-augmented models (Qwen3.8 and GPT5.6) against direct frontier LLM generation, the augmented systems hit 50/50 topology builds and 160/160 intent fields. Direct generation managed only 41/50 builds and 147/160 fields — and that was with the full cumulative scaffold of structure, vocabulary, policy, and container-image guidance still in place. Remove that scaffold and you'd expect worse. Critically, the direct-generation benchmark omitted ScenarioForge's downstream challenge-resource generation, dependency/flag sequencing, deployment, and live validation — the hardest parts. Architecturally, ScenarioForge sits in the constrained-generation family: it doesn't fine-tune an LLM but wraps it in a pipeline of structured constraints — topology templates, service vocabularies, container-image registries, dependency graphs, and runtime validators. The LLM proposes, the pipeline disposes. The emulation substrate is CORE (Common Open Research Emulator), a lightweight network emulator. The key design insight is resource grounding: every service, vulnerability, and artifact in a generated scenario maps to a known-deployable container image or generator, so the LLM cannot hallucinate infrastructure that doesn't exist. The integrity story is mixed. All 226 scenarios validated against the system's own pipeline — the grader and the student are the same team's code. There's no independent red-team audit, no external replication, and no community benchmark for cyber-exercise deployability (because one doesn't exist yet). The 50-prompt benchmark is useful but small, and the comparison to direct generation is somewhat generous to ScenarioForge since the frontier models didn't get the downstream pipeline. The dataset of 226 scenarios and 85 custom generators is a real artifact, but its quality is self-assessed. The milestone question is about scale and adversarial realism. 226 scenarios with 867 steps across 294 Vulhub images is a solid proof of concept, but operational cyber exercises for serious red-team training need multi-stage attack chains with lateral movement, persistence, and realistic defender tooling. The next concrete threshold is something like 50+ multi-stage scenarios with adversarial adaptive scoring validated by external exercise designers — that would demonstrate the system is useful beyond its creators' lab. The obvious missing experiment is adversarial robustness testing: what happens when exercise participants do unexpected things, exploit unintended vulnerabilities in the generated infrastructure, or break the flag-sequencing logic? The authors built and validated the happy path exhaustively but didn't stress-test the scenarios against creative human participants. The honest read: this is likely (a) — they ran out of time/participants. Human-in-the-loop evaluation with actual trainees is expensive and slow, and the system-level validation story was already strong enough for a first paper.
The analogy
Like a home renovation TV host who describes beautiful kitchens but can't produce blueprints a contractor can build from. ScenarioForge is the blueprint layer — it constrains the LLM's fluent descriptions into artifacts that actually compile against real plumbing (container images, network topologies, runtime validators).
Blueprint layer
Translates an architect's vision into buildable plans by constraining every element to real materials, real dimensions, real code compliance
ScenarioForge
Constrains LLM-generated cyber exercises to deployable container images, validated topologies, and runtime-checked flag sequences
Honest assessment
ScenarioForge solves a real and underserved problem — bridging the gap between LLM fluency and deployable cyber-exercise infrastructure — with a clean pipeline architecture and an impressive 226-scenario dataset. The weak point is entirely on the validation side: everything is self-graded, the benchmark comparison structurally favors the system, and no human participants have stress-tested the output. This is a strong engineering contribution and a credible first paper, not yet a proven training tool. Cybersecurity educators and range operators should watch this closely; everyone else can wait for the human-validation follow-up.
What Is CLAIMS?
67
/ 100
BedrockNews applies a structured analytical framework called CLAIMS to academic papers and technical research releases.
How CLAIMS worksClaim
4/5
Ladder
3/5
Architecture
4/5
Integrity
2/5
Milestone
3/5
Successor
4/5
What is this paper actually committing to?
Real gap addressed: LLMs generate fluent exercise text but can't deploy it. ScenarioForge bridges description to deployment. First system in this exact category, though the components (Vulhub, CORE, constrained generation) are all known.
The field fight
Can LLMs generate deployable cybersecurity infrastructure, or only plausible descriptions of it?
Side A
LLMs need heavy constraint scaffolding to produce anything deployable; raw generation is unreliable
Held by: ScenarioForge, constrained-generation advocates
Side B
Frontier LLMs with enough context can generate deployable artifacts directly as models improve
Held by: Direct prompting advocates, scaling believers
Milestone tracker
Unlocks: Confidence that generated exercises withstand creative human participants and can replace manually authored exercises in real training programs
Current
226
scenarios
ScenarioForge dataset
Next
50+
multi-stage scenarios
External human validation
~1-2 years if community adopts and runs adversarial evaluations
Verified Facts (6)
ScenarioForge Execution Dataset contains 226 materialized scenarios covering 294 Vulhub images and all 85 custom-built challenge-resource generators
All 226 passed construction, preview, sequencing, deployment, strict runtime validation, and live artifact checks
867 steps comprising 457 vulnerability-backed and 410 generated-resource-backed instances
ScenarioForge+Qwen3.8 and ScenarioForge+GPT5.6 each achieved 50/50 (100%) topology/build success and 160/160 intent fields
Direct frontier generation reached 41/50 builds and 147/160 fields only with the full cumulative scaffold
The test omitted ScenarioForge's downstream challenge-resource generation, dependency/flag sequencing, deployment, and live validation
Practical Implications (4)
Cyber-range operators could dramatically reduce scenario authoring time if the pipeline generalizes beyond the authors' CORE setup
The 294 Vulhub + 85 custom generator resource vocabulary is itself a reusable artifact for anyone building exercise automation
Model-agnostic design means the pipeline should improve for free as frontier LLMs improve, without retraining
The constrained-generation pattern (vocabulary + runtime validation wrapping an LLM) is applicable far beyond cybersecurity to any domain where LLM output must be deployable
More latest news
Read all