World Editing: Intervening on Executable Worlds at Increasing Depth

We formulate world editing as an intervention on an existing executable world. Let a world be abstractly represented as \(W = (S, T, R, \dots)\), where \(S\) denotes its entities and state space, \(T\) its transition dynamics, and \(R\) the rules and systems that govern interactions. Given a natural-language editing instruction, a world-editing agent transforms \(W\) as

where \(\Delta_k\) denotes an intervention of type \(k\), and \(W'\) is the resulting world that realizes the requested intervention while preserving unrelated properties of \(W\). We characterize interventions by their depth, running from property edits (L1) through entity and dynamics changes to system-level interventions (L4). In game-modding terminology these correspond to parameter, content, mechanic, and system editing.

The levels characterize the semantic scope and coupling of a requested intervention rather than prescribing which implementation components must change. Intervention depth therefore measures how strongly an edit must coordinate multiple parts of the world, not how much code it requires: a system-level edit may be small in implementation yet still need several interacting components to remain consistent.

Two components support this evaluation. IGMWorld provides the executable environment in which agents modify existing game repositories, build and run their edits, and are evaluated on the resulting world rather than on code structure alone. IGMBench provides the task instances: 110 world-editing tasks across Minecraft and Terraria, distributed approximately evenly across the four intervention levels, with 1,161 state and behavioral criteria in total.

Every one of the 1,161 criteria is a deterministic check against the running game. No language or vision–language model is asked to judge whether an edit succeeded. Validators assess observable world behavior rather than source-code structure, so different implementations may satisfy the same intervention, and a task succeeds only if it builds and passes all of its criteria.

The reference library is the game’s own art: 1,989 assets in Minecraft and 15,456 in Terraria. The same construction runs under two text factors, the requested object identity for semantic consistency and “art style” for style consistency, so “consistent” means as close to the game’s art as the game’s art is to itself. We checked the choices for stability (k from 1 to 10 preserves at least 88% of pass/fail verdicts) and against people: nine raters with prior gaming experience judged 40 blind-sampled agent textures, and the style factor agrees with the majority human judgment at 0.66 AUC, against 0.54 for a CSD style embedding.

Here \(x\) is the evaluated asset, \(I_c\) the native reference assets of category \(c\), and \(f\) the text factor. An asset passes when \(S_f(x;c) \ge \tau_{c,f}\), where the threshold \(\tau_{c,f}\) is the \(\alpha\)-quantile of leave-one-out scores over \(I_c\).

Current frontier agents already exhibit substantial functional world-editing capability. Criterion Pass Rate runs well above the stricter World-Editing Success Rate throughout, indicating that many unsuccessful tasks nevertheless satisfy substantial portions of the requested behavior.

Reliability varies systematically with intervention depth. Every evaluated configuration achieves its highest WSR at L1, while performance is generally lower for deeper dynamics and system interventions, although the relative ordering of L3 and L4 varies across models and games. Because deeper tasks also contain more evaluation criteria, we stratify tasks by criterion count: within comparable criterion-count ranges, L1 remains substantially more reliable, suggesting that intervention depth captures structure beyond criterion count alone.

Most unsuccessful world edits fail after reaching an executable state. Behavioral failures account for the large majority of unsuccessful runs at every intervention level, while build and load failures remain comparatively uncommon. Even at L4, where build failures become more frequent, behavioral correctness remains the dominant failure stage. The principal bottleneck is therefore not producing a modification that compiles and loads, but realizing the requested behavior in the executable world.

The composition of behavioral failures also changes with depth. L1 failures are almost entirely mechanic semantics, such as a wrong value or formula. From L2 onward, interaction/progression failures, in which new content misbehaves on contact with existing systems, become the largest category and keep growing with depth, while resource/registration failures stay flat. This is consistent with deeper edits requiring coordination across multiple interacting world components rather than a single localized change.

Visual integration remains substantially weaker than functional world editing. In both games, native assets clear the joint style-and-semantic check at a rate no evaluated agent approaches.

Visual performance does not follow the ranking observed for functional world editing. Configurations that are weaker on executable state and behavioral correctness can nevertheless perform competitively on visual assets, while the strongest functional agents do not dominate the visual evaluation. This weak correspondence suggests that world editing is multidimensional: successfully modifying the behavior of an executable world does not imply that newly introduced content integrates perceptually with that world.

Although IGMBench quantitatively evaluates Minecraft and Terraria, the formulation is not specific to these games. World editing is better viewed as a general capability over executable worlds, with games serving as a concrete and verifiable testbed rather than the boundary of the problem.