anteloc/ldraw-nova: Agent tooling for generative LEGO fashions constructing, constructed with Astra and Opus 5.5, powered by Jev · GitHub

Give an AI agent a mannequin thought. Guide it, let it construct it, and get an LDraw LEGO© mannequin.

What you get when the constructing course of finishes:

• Its supply code, in LDraw language.
• Different views: 3D viewer, 3D participant, VR interactive (Meta Quest 3), photographs…
• Blender editable glTF file, in .glb format, metainfo as Blender’s Custom Properties.
• Chat historical past and agent considering course of.
• and extra… 👌

Take a have a look at the video:

Run ldraw-nova as an online app, with Docker. You want Git and Docker.

This internet app will run dockerized, and to construct the Docker picture, two sibling repos are required:

1. Clone each repos facet by facet, on the similar tag, so that they work collectively:

git clone --branch v0.6.0 https://github.com/anteloc/ldraw-nova.git
git clone --branch v0.6.0 https://github.com/anteloc/ldraw-nova-docker.git

2. Build the Docker picture. The first construct takes some time and desires about 5 GB of disk house:

cd ldraw-nova-docker
docker compose construct

3. Start the app:

4. Open it in your browser:

• https://localhost:8443: wanted for VR on Meta Quest 3. The certificates is self-signed, so settle for the browser’s warning the primary time.
• http://localhost:8765: plain HTTP, no certificates warnings. Use it if the self-signed certificates will get in the best way. VR will not work over it.

Other units in your community can attain the app by your laptop’s IP as a substitute of localhost, e.g. https://192.168.1.20:8443 from a Quest 3. The app has no login, so solely run it on networks you belief.

To cease it:

Well, to summarize: I did this in an effort to get agentic LLMs able to designing buildable, bodily issues!

Finding LDraw, an meeting language (pun meant! 😜) that will be on the similar time easy, low degree, and executable in an effort to produce 3D CAD fashions, gave me the thought of experimenting with each ChatGPT and Claude in an effort to try to make them code in LDraw, similar as they do with different programming languages.

To my shock, though this language is closely targeted on math (components rotations, positioning…), which LLMs are normally unhealthy at, brokers did fairly effectively as a substitute on preliminary assessments, and subsequent initiatives additionally yielded good outcomes, however by no means sufficient in an effort to take into account generated fashions to be right:

These three makes an attempt, and fairly another experimentation, led me to the next conclusions:

💡 Conclusion 1: there’s a minimal resistance path to geometry math for brokers, i.e.:

• Giving the brokers tooling to generate LDraw sources would sidestep (evil!) geometry math
• … as a result of they do manner higher at producing python code that produces math
• … than on producing math themselves!

💡 Conclusion 2:

• Agents are inclined to do higher when studying from python code that produces fashions
• … than from fashions themselves (LDraw’s evil geometry, once more…)

Then, the solely factor left 🤔 was to create a python-based tooling with the required primitives, verbs, constructive vocabulary… so brokers would be taught by instance and do comparable issues on their very own.

Which proved to be actually arduous to get proper, even when vibe coding it… till GPT-6 Astra and Claude Opus 5.5 arrived… and vibe-coded it proper! 🚀🚀🚀

ldraw-nova supplies the instruments, examples and directions an agent must design fashions with actual LDraw components.

The course of is as follows:

1. The agent takes a immediate.
2. Reads instructions.md and associated paperwork to LDraw language and LEGO© fashions constructing.
3. Plans the best way to construct the mannequin: required components, submodels to be created, aesthetics…
4. Iteratively:
   1. Renders photographs from the mannequin/submodel(s)
   2. Inspects them, adjusts positioning, aesthetics… and again to rendering
5. … till it considers the mannequin completed and able to ship!

Provided tooling helps the agent in:

• Finding appropriate components.
• Also, instance fashions and submodels to begin with.
• Collision and gaps detection for putting components accurately.
• Headless rendering for inspecting present outcomes.
• and extra…

The agent would not truly begin with putting components, apart from issues like e.g. prototyping and studying by altering pre-existing instance fashions.

The manner it produces fashions is extra like:

• Collects the required data, from experimental outcomes, docs and planning.
• Builds a number of plans, that absolutely describe the mannequin and submodels, together with its geometry, like e.g. atlas-crane.plan.json
• And with that plan, it creates a number of generator scripts like e.g. generate.py
• … that when executed, produce LDraw supply file(s), a really specialised 3D CAD language.
• … like e.g. atlas-crane.mpd

To summarize, that is like:

• an agent making a generator
• … that produces a 3D mannequin
• … in an meeting language named LDraw 🤯

A compiler of types, so to say 🤓

flowchart TD
    agent([agent]) -- produces --> plan[plan.json]
    plan -- interpretation --> gen[generator.py]
    gen -- execution --> mannequin[model.mpd]

These are a few of the guides and references given to the agent in an effort to make it a builder:

Being this a first launch, there are fairly some issues that also require some work:

• VR on Meta Quest 3: mannequin dealing with has many points, efficiency points.
• Adapt for low-end brokers: adapt present tooling, docs and directions in an effort to enhance utilization by low-end fashions like e.g. Luna, Haiku, and so on.
• Expensive technology: presently, solely costly, high-end fashions, are presently able to producing large-sized and proper fashions.
• Improve effectivity: generative course of is presently gradual.
• Add and enhance extra mannequin households:
  • Humans and animals: minifigs
  • Technic fashions: machines, engines…
  • Spaceships: generated fashions usually are not superb
• Building fashions from manuals: it partially works, higher if web page manuals are given as photographs.
• Fine-grained inspection: for inspecting submodels and their step-by-step constructing processes.

COMING SOON

I’d wish to thank the next:

• The LDraw Community
• LDView‘s Travis Cobbs (@tcobbs), and contributors.
• LeoCAD‘s Leonardo Zide (@leozide), and contributors.
• LDCad and Shadow Library, Roland Melkert.
• ldraw.rs‘s Park Joon-Kyu (@segfault87), and contributors.
• pyldraw3‘s Harold Martin (@hbmartin), and contributors.

… and due to all the many different LDraw creators!

NOTE: For this work, I’ve used many LDraw fashions, libraries, instruments, docs… from many sources.

There is rather a lot wonderful folks that generously contributed to this, even for many years, by generously donating their most interesting work to the general public area and open supply neighborhood.

If you suppose you have to be included on this part, please drop me an electronic mail!

LEGO(R) is a trademark of the LEGO Group of corporations which doesn’t sponsor, authorize or endorse this software program.