Controllable Generative Modeling in Minecraft by Training on Billions of Cubes

This page requires JavaScript to render the full Distill layout.

We introduce Dream-Cubed, a new large-scale dataset and family of generative models for generating Minecraft worlds at block resolution. Our data comprises billions of high-quality and carefully-balanced cubes from procedurally generated Minecraft terrain and human-authored maps, which we use to study discrete and continuous 3D diffusion models for biome-conditioned chunk generation. When trained with our data, we show that both approaches can generate high-fidelity chunks of the game world, and that the discrete masked diffusion formulation gives us inpainting, outpaining, and user-defined block conditioned generation as a free byproduct of the training objective. This enables players and creators to mold the world around them by generating structures, terrain, and maps that are immediately editable and playable.

Generative AI has made incredible progress in the fields of image, video, and text generation. Despite success in these modalities, the interactive 3D worlds of video games have received much less research attention. We aim to close this gap by releasing a dataset based on one of the most successful and popular video games, Minecraft.

Minecraft is the best-selling video game in history, and a favorite testbed for a variety of AI research. The 3D world of Minecraft is made up of voxels: discrete, compositional building blocks like sand, stone, and water. These semantically meaningful block types provide a perfect abstraction for generative modeling: each piece of the game world is simply a collection of block tokens that come together to create a simulacrum of the natural world. These collections of blocks, or "chunks", capture anything from winding rivers and snow-capped mountains to medieval city centers and futuristic sci-fi worlds. The compact block-based abstraction and nearly infinite expressive range make Minecraft an attractive substrate to study generative systems.

Central to Minecraft is its procedural content generation system, which generates a massive, unique world for each player. These worlds are split into biomes like desert, forest, and ocean, creating a diverse set of environments to explore. Players interact with the world by destroying and placing blocks, creating structures, and terraforming the world as they play. We seek to explore both sides of Minecraft in our work: the natural world generated by algorithmic systems, and human-built structures that capture human creativity in this voxel world.

The Dream-Cubed dataset contains both procedurally generated and human-authored Minecraft chunks. We create automated data collection and curation pipelines, collecting 1.8+ million 323 chunks of procedurally generated Minecraft worlds, and 200,00+ chunks from human-authored maps, comprising over 30 billion total blocks.

Minecraft's core procedural content generation provides an endless supply of data. We collect chunks from the surface level of the world and assign a class label based on the biome they are in, creating a dataset for biome-conditional generation. While this data is functionally infinite, biomes are not evenly distributed in generated worlds. Villages, for example, are both the most structurally complex and the least common areas of the game world. To overcome this data balancing problem, we set up targeted collection pipelines to identify and collect specific biomes to ensure adequate representation, which we show can have a very significant impact on downstream models.

Human data, on the other hand, is scarce. There is a rich community centered around the content that players create in their Minecraft worlds. Popular third-party websites like Planet Minecraft allow users to share their creations with others, and some professionals earn income by selling their hand-crafted maps. These maps pose a much deeper challenge for generative modeling, using a larger vocabulary of blocks and containing much more complex structures and environments. We reached out to one such professional map creator, who granted permission to use several of their large and high-quality maps for scraping. We release all our data and models in accordance with the map creators for the larger research and Minecraft community.

Each Minecraft block is categorical, representing a single block type from a vocabulary of thousands. A natural question is: what kind of generative model is best suited for this data? Continuous diffusion models like DDPM

To understand what makes a great foundation model for Minecraft, we study both diffusion formulations, using an identical Diffusion Transformer backbone

A key design decision is not to compress the spatial dimensions of our data. Commonly, diffusion models operate on the compressed representations from models like Variational Autoencoders, which can massively improve the efficiency of diffusion models. We choose to train directly at the native block resolution, feeding each 323 chunk of the world into a 3D Diffusion Transformer with ~280M parameters. While we miss out on the performance gains of compressed representations, we find this choice is crucial for enabling fine-grained control over generation.

We implement two diffusion formulations:

With our carefully curated data and after hyper-parameter tuning, we show that both models perform well on the training task, generating believable 323 conditioned on a target biome, as seen below. While this doesn't tell us which formulation is better for Minecraft, we find the discrete diffusion formulation unlocks a variety of interactive generation workflows beyond the single-chunk generation case, making it a more compelling choice for an interactive world model.

The combination of modeling our data with discrete diffusion and training at the full spatial resolution unlocks some immediate and compelling use cases for our models.

MD4 is a type of masked diffusion, where the forward process iteratively replaces blocks with [MASK] tokens. Then, in the reverse process, each [MASK] token is iteratively unmasked using the predicted block to condition the rest of the sampling process. This simple property gives us a fine-grained block-conditioning on top of class-conditioning as a free byproduct of training.

In practice, this means we can write any pattern of blocks as an initial state and even during the generation process, which becomes a hard constraint that the model must work around. Because we work at the native resolution using the in-game block vocabulary, this translates to simply placing a handful of Minecraft blocks in a desired pattern. This lets us interface directly with the game's content: you can regenerate parts of a chunk taken directly from the game world, or hand-craft custom patterns, then let the model fill in the rest.

What does this look like in practice? First, we can simply create a custom "seed" pattern of blocks, representing things that are rare or non-existent in the procedurally generated worlds the model was trained on. We can make spirals, waves, and slopes of blocks to see how our model works with these constraints to create a realistic chunk of the game world. Even suggestive patterns like a volcano, which don't naturally occur in Minecraft, are no problem and result in successful extrapolation. We show some seed patterns and the resulting inpainted chunks below:

This property lets us use inpainting for conditional generation on a 323 chunk basis, but we can take it a step further. By leveraging this property for outpainting, we're no longer stuck generating small 323 chunks of the world. We can use a simple sliding window procedure to generate arbitrarily-sized worlds, generating overlapping chunks in sequence and stitching them together:

Visualized below are a few examples of larger world chunks generated this way. The first 3 worlds are generated without biome conditions, creating worlds guided only by block contexts. The final sample incorporates biome conditioning, showing how the biome-conditional model can be leveraged to guide generation towards desired terrain features.

This outpainting method leverages the same "fixed-block" property but is limited to slices of neighboring chunks rather than to user-defined block patterns. These methods are not mutually exclusive, which means we can push sampling even further. We design seed block patterns, similar to the ones we used for 323 chunks, but spanning hundreds of blocks. We slice these into overlapping 323 chunks, and then provide both the seed pattern and neighboring chunk contexts to large worlds with hand-crafted terrain features.

We show some examples of this process below. The "Show Seed Context" button will display the block pattern used to guide generation.

When training on human-authored maps, we treat each map as its own additional "biome", and compress the thousands of block types present in this data down to a smaller, more manageable vocabulary of 171 unique blocks. As map creators use a much larger palette of blocks to capture small visual details, compressing maps to a limited vocabulary results in data that is simpler, allowing our models to focus instead on capturing the complex and detailed human structures. We acknowledge that learning to generate from data that encodes human creativity comes with a new set of challenges and even ethical considerations, and we present our models and results as a first step in this domain.

Lacking established metrics for 3D voxel worlds, we adapt Fréchet Inception Distance (FID)

To test how important data composition is, we create three variants of our dataset, each with 1 million chunks. The Balanced dataset represents even distribution between the 15 natural biome classes, while the Boosted dataset doubles the proportion of village samples. The Natural dataset represents the naive approach, collecting chunks at their natural spawn rate in the game world, leading to large imbalances between biome labels.

The average FID across all biomes is roughly equivalent between all of our models, though we find some notable differences within biomes. We can see a significant performance drop in our patch 4 models in the village biome, suggesting that the finer resolution at lower patch sizes is important for modeling this more complex data. Similarly, we see differences regarding dataset composition: the Natural dataset model performs well on the overrepresented ocean biome (35% of the dataset), and underperforms in the village biome (< 1% of the dataset). We find that doubling the number of village samples compared to the rest in the Boosted model, we see a significant improvement in FID.

While the biome-conditional samples we generate look convincing at a glance, how similar are they to the real world of Minecraft? We turn to the experts on this 3D voxel world: Minecraft players.

We create an experiment where humans view two sets of chunks for the same biome, and pick the one they think is better. We include samples generated by two of our discrete models, our DDPM model, and real samples taken from our training data. This lets us directly test whether our generated chunks are convincing, by analyzing how often they were preferred to real Minecraft world chunks.

Across all three models, generated chunks were actually preferred to real ones. This may be partly due to our process of data curation and generation, but we view it as evidence that generated chunks are at least on par with the distribution we're trying to model.

Using this data, we validate the render-based FID metric we use. If image-based FID is a good proxy for 3D chunk quality, we'd expect that lower-FID models would win more against higher FID models.

| Min FID gap | Agreement | n | p | |---|---|---|---| | 0 | 54.3% | 512 | 0.0286 | | 5 | 61.7% | 227 | < 0.001 | | 10 | 62.9% | 159 | < 0.001 | | 15 | 66.1% | 109 | < 0.001 |

And we do find a limited but significant relationship between FID and win rate. In particular, we find that for biomes where there are small FID differences across models, this relationship is weaker. However, focusing on more structured biomes where the FID differences are larger, alignment of our metric with human judgment becomes stronger, with the lower-FID model winning 66.1% of the time when the gap is > 15 points.

Minecraft is a rich domain for studying 3D generative models, and we hope to enable and accelerate future efforts by releasing the Dream-Cubed dataset with millions of samples to learn from. We conduct an initial exploration in this space, studying block-level diffusion models in both discrete and continuous formulations. While both approaches work, we dig into some unique properties of discrete masked diffusion, showing how controllability emerges naturally from the training objective, taking us from 323 class-conditional chunk generation to infinitely sized worlds with fine-grained block-level control. We refer to our technical report for all additional details, and release the full extent of our collected data, code, and model weights for the wider research and Minecraft community.

We release the paper, code, model artifacts, and datasets to support future work on generative modeling for interactive 3D worlds.

To cite our work or this blog post, you can use the following BibTeX entry:

@article{merino2026dream3,
  title={Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes},
  author={Merino, Tim and Earle, Sam and Iwai, Ryunosuke and Togelius, Julian and Cetin, Edoardo},
  journal={arXiv preprint arXiv:2604.22847},
  year={2026}
}