Games Must Be Played: Oasis and the Next Benchmark for Game Development Agents

Games Must Be Played

A game is not a screenshot. A game is not a repository. A game is not a short video that looks convincing for twenty seconds. A game is an interactive system: scenes, assets, physics, animation, navigation, UI, camera, state machines, rules, feedback, and player action all running together inside an engine.

That is why game-development agents need a different kind of benchmark.

The frontier is no longer just asking whether an agent can edit a script or assemble a toy project. The frontier is whether an agent can understand an engine-native world, modify it in 2D and 3D space, wire gameplay systems, preserve runtime invariants, and produce something that can actually be played. The reliable way to evaluate that capability is to play the result under controlled conditions and observe what the engine knows to be true.

This is the principle behind Oasis: games must be played to be evaluated.

Oasis is not designed as another single-score leaderboard. It is a game-agent evaluation harness built around a more precise question: which parts of game development does an agent understand, and which parts fail when the game is exercised inside a real engine? The answer requires capability trees, deterministic PlayTests, runtime telemetry, and intelligent judgment grounded in evidence.

Where Current Benchmarks Fall Short

Recent benchmarks have made game-agent evaluation serious. GameDevBench places agents inside real Godot tasks and evaluates their changes with hidden validation. GameCraft-Bench moves toward end-to-end playable projects, replay traces, video evidence, and multimodal judging. Both are important steps. They also reveal why the next benchmark has to be more engine-grounded, spatial, and play-centered.

GameDevBench already shows why this direction matters. Under automated validation, Seele02-pro reaches 209/333, 62.8% Pass@1, while Seele02-flash reaches 183/333, 55.0% Pass@1. Pro improves over Flash by +26 tasks and +7.8 percentage points, showing a clear gain in engine-grounded game-development execution under the same SeeleAgent harness.

The Pro lift is not just a leaderboard delta. It suggests stronger multi-step engineering, requirement retention, validation-oriented convergence, and routine Godot structure completion. At the same time, the remaining failures point to a harder frontier: precise geometry, spatial target localization, Godot resource chains, runtime interaction, and long-chain self-validation.

That is exactly why Oasis is organized as a capability tree instead of a single number, and why future evaluation must include runtime PlayTests rather than relying only on hidden assertions or surface-level demos. If an agent fails because it misreads a coordinate frame, leaves a ShaderMaterial chain incomplete, or creates a UI interaction that only breaks at runtime, the benchmark should report that leaf capability directly.

GameCraft-Bench provides the complementary end-to-end view. On the GameCraft-Bench result page, Seele02-pro tops GameCraft-Bench at 50.7%, with 71.0 / 140 raw score sum, 140 / 140 cases scored, and a best single-case score of 87.2%. That is +9.2 points over Opus-4.7 high at 41.5%.

Together, the two benchmarks show different axes of game-agent capability. GameDevBench highlights engine-grounded issue solving under hidden validation: can the agent repair a real Godot project so the verifier passes? GameCraft-Bench highlights playable end-to-end generation: can the agent ship a complete runnable project that produces replay evidence and earns a high rubric score?

Oasis is designed to measure both axes while adding what neither benchmark fully captures: runtime-truth PlayTests, first-class spatial diagnostics, and capability-tree reporting. The difference is not that earlier benchmarks are wrong. The difference is that game development is broader than either hidden code contracts or video scoring. A benchmark for the Agent Harness era has to measure the whole loop: build, play, observe, diagnose, and improve.

The Oasis View: A Capability Tree for Game Development Agents

Game development is not one skill. An agent can understand scripting but fail at scene graphs. It can generate assets but place them at the wrong scale. It can create a beautiful room with no valid navigation path. It can pass a unit test and still softlock a player after three interactions.

Oasis therefore starts with a capability tree. The benchmark is organized around leaf capabilities, and each leaf gets a verifier appropriate to what it measures. Some leaves need hidden engine tests. Some need runtime invariants. Some need deterministic PlayTests. Some need AI-assisted judgment for semantic fit, visual readability, and style coherence.

This structure matters because a single score cannot explain game-development ability. Oasis reports capability-domain scores and leaf-node failures so that researchers can see whether a model is failing at engine semantics, 3D spatial editing, gameplay wiring, asset integration, robustness, or end-to-end assembly.

The benchmark also includes public scaffolds: text-to-image asset generation, image-to-3D pipelines, engine-native Playgrounds, runtime telemetry APIs, and reusable scenario templates. The goal is not to reduce creativity, but to make it measurable inside a playable world.

A Five-Layer Evaluation Stack

The design behind Oasis separates strict public contracts from flexible implementation choices. Hidden tests should hide seeds, scenarios, and thresholds, not undocumented assumptions about one exact node path or serialized resource shape. The resulting stack moves from basic engine readiness to semantic evaluation.

Playground-Based PlayTest

The missing primitive in game-agent evaluation is PlayTest.

In Oasis, a Playground is a prebuilt engine-native environment with modules, state observation, action interfaces, verification hooks, and task-specific scenarios. The agent implements or modifies a feature, level, object, mechanic, quest, system, or 3D space inside that Playground. The benchmark then runs a deterministic PlayTest to determine whether the modification works through legal interaction.

AI does not play the game. The harness plays the game.

That distinction is important. Oasis does not train an AI player to control keyboard and mouse and then mix player ability into developer evaluation. Instead, the PlayTest is executed by predefined scenarios, replay traces, or domain-specific action sequences. These actions can move, jump, attack, interact, open menus, pick up items, trigger dialogue, or wait for events. They cannot simply set a victory flag, teleport a player to the endpoint, zero out a boss health variable, or inject an item directly into inventory.

The analogy is embodied simulation. In a robot manipulation benchmark, a system should not be allowed to declare success by setting an object's xyz position directly. It has to act through the allowed control space. Oasis applies the same idea to game development: the game must be solved through the game, not by editing the answer into the state.

This is where SeeleAI's view of game creation becomes specific. Game creation is not prompt-to-demo. It is engine-native playable world editing. A capable agent needs to understand scene structure, runtime state, 3D layout, collision, navigation, visual affordances, and the relationship between player action and world response.

Runtime Truth Over Blind Video Judgment

Video is useful evidence, but video is not truth. The engine already knows far more than a video can show.

During a PlayTest, the harness can observe player transforms, velocities, collision bodies, overlap events, navmesh paths, enemy states, health values, inventory, quest flags, dialogue states, animation states, UI state, trigger events, timing, softlocks, out-of-bounds positions, and invariant violations. This runtime truth is the strongest source of objective evaluation.

Consider a collision bug. A video judge might notice that a character appears to pass through a wall if the moment is visible and unambiguous. But if the penetration lasts only a few frames, happens behind the camera, or occurs in a branch not shown by the demo, the judge may miss it. A runtime invariant can report that the player collider overlapped a solid volume during a non-phasing state, that a navigation agent crossed an invalid region, or that a quest completed without the required event chain.

The engine already knows the truth. The benchmark should use that truth directly.

This makes PlayTest stronger than static tests and blind video judgment. Static tests can confirm that a resource exists or an API returns a value. Video can show that something looks plausible. Runtime telemetry can show that the world responded correctly under controlled play.

AI-Assisted Judgment, Not AI-Only Scoring

Oasis is not anti-AI. It is against using AI as the only source of truth.

Many game qualities benefit from intelligent judgment: whether a 3D asset semantically matches its role, whether UI is readable, whether a level communicates direction, whether feedback is understandable, whether an art style is coherent, whether dialogue makes sense, or whether a mechanic expresses the design intent. These are not always reducible to a single mechanical assertion.

But AI judgment should sit on top of runtime evidence, not replace it. In Oasis, AI can be introduced at the Judge layer as an evidence assistant. The judge can receive screenshots, replay clips, event logs, trajectory summaries, invariant reports, state snapshots, and task requirements, then evaluate semantic and aesthetic dimensions that are hard to capture with rules alone.

Core acceptance remains grounded in the harness: build success, scenario completion, legal interaction, collision validity, navigation reachability, inventory state, quest progression, softlock detection, and replay determinism. AI can score higher-level meaning, but it should not be asked to guess whether a game worked from pixels alone.

This is the difference between AI-only scoring and AI-assisted evaluation. The former asks a model to be the judge of record. The latter gives the model evidence produced by an engine-aware verifier.

Beyond a Single Score

Oasis has reached its release-ready form and is planned for public release soon. The public release will not be framed as a single opaque number. It will publish an evaluation protocol and evidence-backed report format that show what happened inside the engine, which capability leaves were exercised, and why a run succeeded or failed.

The point is not only to rank agents. It is to make game-development ability legible: where the agent understands the engine, where it breaks in 3D space, where it passes a demo path but fails a controlled PlayTest, and where semantic judgment should be grounded in evidence rather than pixels alone.

Toward an Open Benchmark for Playable Worlds

SeeleAI plans to release Oasis publicly to support a more rigorous standard for game-development agents. The benchmark is intended for researchers, engine-tool builders, model developers, and creators who need to evaluate more than compilation or video appeal.

The larger goal is to move game-agent evaluation into the Agent Harness era. In that era, agents are evaluated inside worlds that can be played, observed, stress-tested, and improved. Their outputs are measured by engine semantics, 3D spatial consistency, gameplay behavior, runtime robustness, and semantic quality.

This is also the standard SeeleAI is building toward: not generic game-like generation, but engine-native understanding, runtime-aware editing, and 3D playable world construction. A game-development agent should know how the world is built, played, broken, and fixed.

The future of AI game creation is not prompt-to-demo. It is harness-grounded game development: build, play, observe, and improve.