Blender as API

ChatGPT screenshot

I heard GPT6 Astra has strong capability in 3D related tasks. I saw a tweet of my friend, showing mulitple VJ clips created by Astra with Blender, instead of let GenAIs generate video clips directly:

It makes more sense to me than just generating videos directly with Nano Banana etc; it gives us more control over the entire video generation workflow. Astra generates a Python script as a 3D scene setup, easy to version control. We can edit the materials on Blender, then let Astra reuse it to generate a variety of video clips with a consistent visual taste.

image

First output

First I started with a no-brainer prompt: "Create a cool VJ clip on Blender". Ofc the result was terrible, but at least I could confirm that the Astra + Blender workflow actually does the job.

Second I gave it some images as reference and said "Create geometric scene with metallic materials like". Astra made clips kinda okay, but it lacked the sense of color, post-processing and details. However after I threw more requests like "Add bloom, color lights and aberration" it made some progress, it was convincing to me that we can improve the output by telling what pleases our eyes betterf, just like when we do UI dev.

Alloy_look_comparison

Then I remembered Blender 5.3 added dispersion support (rainbow-ish refaction of transparent materials like glass, water etc).

I asked him to provide me a list of ideas with test render. It was surprising that it made a variety of scenes, though some of them didn't make sense as VJ clip.

image

I picked some of them, added some ideas, and let it proceed to full render. Yet they are too lame to be used for VJ directly, they look just a few steps away from what I'd say it's finished. You know it's AI, the cold bloodless thing, it always try to use dumb simple shapes like square, triangle etc. In the glass flacture sample, all the fractures were strait-cut triangles. It took 20~30 iterations to make the shapes naturalistic, finess the lights and materials, adjust the post-effects.

The final outputs were like this:

To be accurate, I've seen many folks trying to control Blender with AI since GPT-3.5 era. But thanks to the fast iteration & strong compture-use ability in Astra, it's like Blender is acting like a remote renderer API, kinda more purified & abstracted than working as an MCP server.

However there's still some quirks; it can't actually judge the visual quality of the output by itself. It knows what is thought to be "cool" as a knowledge, it knows what pleases humans' eyes, but its doesn't feel pleasure. At least, not yet.

For example, I once tried to generate a particle scene. The sketche generated by GPT Image was good, but it failed to reproduce it in Blender. He understands the base shape, but can't reason how those particles should move, unlike we can easily imagine from the image. The output was just a wobbling sphere, without the flare-like flows in the sketch.

image

Once manage to tame Astra completely to let it generate a perfect scene, the bottleneck moves from manipulating Blender to:

  1. Explain your demand in format AI can read
  2. Rendering time

1. is　the problem that we're all facing in software development. But in visual development it could be even more difficult, since it's not straightforward to translate the image in our brain into text. 2. is a hard problem. It solely depends on the path tracing algorithm and the GPU power. We can use faster renderer like EEVEE or choose game engines like Unity / UE, but if you need high quality outputs, it's an necessarry expense——until AGI discovers a blazing fast pathtracing technique.