LegoGPT generates a LEGO structure from a user-provided text prompt in an end-to-end manner. Notably, our generated LEGO structure is physically stable and buildable.
LegoGPT generates a LEGO structure from a user-provided text prompt in an end-to-end manner. Notably, our generated LEGO structure is physically stable and buildable.
We introduce LegoGPT, the first approach for generating physically stable LEGO brick models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of LEGO designs, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that LegoGPT produces stable, diverse, and aesthetically pleasing LEGO designs that align closely with the input text prompts. We also develop a text-based LEGO texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We also release our new dataset, StableText2Lego, containing over 47,000 LEGO structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models.
(a) From a ShapeNetCore mesh, we generate a LEGO design by voxelizing it onto a \(20\times 20 \times 20\) grid and applying legolization to determine the brick layout. (b) We augment each shape with multiple structural variations by randomizing the brick layout while preserving the overall shape. (c) Stability analysis is performed on each variation to filter out physically unstable designs. (d) To obtain the corresponding captions for each shape, we render the LEGO design from 24 different viewpoints and use GPT-4o to generate detailed geometric descriptions. (e) Data samples from 5 categories in our StableText2Lego dataset.
(a) Our system tokenizes a LEGO design into a sequence of text tokens, ordered in a raster-scan manner from bottom to top. (b) We create an instruction dataset pairing brick sequences with descriptions to fine-tune LLaMA-3.2-Instruct-1B. (c) At inference time, LegoGPT generates LEGO designs incrementally by predicting one brick at a time given a text prompt. For each generated brick, we perform validity checks to ensure it is well-formatted, exists in our brick library, and does not collide with existing bricks. After completing the design, we verify its physical stability. If the structure is unstable, we roll back to the stable state by removing all the unstable bricks and their subsequent ones and resume generation from that point.
"A streamlined vessel with a long, narrow hull"
"A classical guitar"
"A basic sofa"
"A bookshelf with horizontal tiers"
"A high-backed chair"
"A backless bench with armrest"
"A streamlined vessel with a long, narrow hull [...]"
"An asymmetrical six-string guitar [...]"
"Rustic stone bench with moss growth [...]"
"Hot rod with flame paintwork [...]"
"Rustic farmhouse chair built from reclaimed wood [...]"
"Live edge walnut table [...]"
"Comfortable lounge chair wrapped in Japanese shibori fabric [...]"
"Cyberpunk holographic material with neon purple and blue gradients [...]"
"Rustic farmhouse armchair built from reclaimed wood [...]"
"Vintage floral tapestry with deep reds and golds [...]"
"Gothic cathedral bookshelf with arch details, medieval style [...]"
"Japanese sliding bookcase with shoji screens, traditional design [...]"
"Victorian library shelving with carved moldings [...]"
"Parlor guitar with ladder bracing [...]"
"Electric guitar in metallic purple [...]"
"Steel resonator with engraved body[...]"
"Sunburst Les Paul with amber finish [...]"
@article{pun2025legogpt,
title = {Generating Physically Stable and Buildable LEGO Designs from Text},
author = {Pun, Ava and Deng, Kangle and Liu, Ruixuan and Ramanan, Deva and Liu, Changliu and Zhu, Jun-Yan},
journal = {arXiv preprint arXiv:2505.05469},
year = {2025}
}
We thank Minchen Li, Ken Goldberg, Nupur Kumari, Ruihan Gao, and Yihao Shi for their discussions and help. We also thank Jiaoyang Li, Philip Huang, and Shobhit Aggarwal for developing the bimanual robotic system. This work is partly supported by the Packard Foundation, Cisco Research Grant, and Amazon Faculty Award. This work is also in part supported by the Manufacturing Futures Institute, Carnegie Mellon University, through a grant from the Richard King Mellon Foundation. KD is supported by the Microsoft Research PhD Fellowship. The website template is taken from Custom Diffusion (which was built on DreamFusion's project page).