in Lotusland

Jev VJ

Wed, 23 Sep 2026 00:00:00 GMT

Like everyone else on the Internet, I also created a Jev demo: a VJ app with Jev.

I made a #Jev VJ demo.

Jev listens to what I say, suggests the matching video clips to me, then I manually switch them.

I wrote a list of clips & effects with short description beforehand, so it can pick the best combination, just like the official "What color is the sky" demo. https://t.co/0YVfoa6INP pic.twitter.com/4S5SoDgoEJ
— 𝘼𝙈𝘼𝙂𝙄 (@amagitakayosi) September 22, 2026

Code: https://github.com/fand/jev-vj

In this demo, Jev works as a companion. To the user prompt in the textbox at the bottom, Jev picks the optimal combinations of clip & effect. User switches them manually, so the videos actually play perfectly in sync with the music.

I'll explain what I've found while building this, and my thoughts on AI-aided visual development.

Architecture

So like you all, I just stumbled upon the post by Diogo. I found it interesting, especially Jev's ability to play Doom. I'm a graphics nerd and I'm always looking for new ways for VJ-ing, and VJ-ing is kinda similar to playing video games, I thought Jev can lead us to a new VJ experience.

But soon I hit a wall. Jev's gameplay ability is based on its Choice model, which excels at finding the optimal answer from the provided choices. To put this into practice we need to provide the current game state, in other words the state must be evaluable. However in VJ performance the evaluation is our perception, there's no score.

Also, even with Jev's blazing-fast response we cannot let it switch the videos exactly on the beat. Here in Vancouver, Jev's response time is around 90ms-260ms, which is roughly one or two 16th notes at 120BPM, still too much latency for music / live visuals.

So I changed the approach; I let Jev just suggest candidates for the next clip. I created a full clip list with metadata like color, bpm, camera movement etc, and included it in the context of the prompt. Jev uses it as the criteria to score the choices and suggest clips in async, then we evaluate the suggested clips and switch them on the beat. The latency doesn't matter anymore.

I let Astra create a demo with this approach. It required a few hours of iteration to stabilize the UI and fix bugs / performance issues, but finally it worked!

Clip Library

The biggest problems heres is obviously the library. We have to provide a thorough list of the clips with descriptions concise yet detailed enough to let Jev find the best clips. Unfortunately I couldn't find a way to fully automate this preparation. Here's what I tried:

First i tried extracting every 10th frames from a video with ffmpeg, then fed them to Astra. It comprehends static images pretty well, but Astra cannot understand the motion inside the frame sequence. I also tried increasing frame frequency but it didn't help.

I heard Gemini can handle videos pretty well, so I tried it next. At first it looked promising; Gemini gave me a full list of features in the video, like color palette, materials, and the camera movement. However I found that it cannot remember the details when there are many moving objects in the video. One example is this video by Mantissa; there are thousands of particles moving rapidly and a hard light flicker, but Gemini described this clip "ideal for slow ambient music" just looking at the camera movement speed.

For this demo, I took a semi-manual approach. I let Astra prepare a Markdown template with filenames, then I wrote description for every video using AquaVoice. This was tedious but most flexible as we can edit it easily 😇

Later I made a library editor, where we can manage those clip data in a spreadsheet. It's still tedious... but I think we need this kind of manual library work for any kind of VJ work, even with completely different AI-aided systems.

Token limit

Once you got a full detailed clip list, it leads to another problem; token limit. With 112 clips I used, the token count was 27k, 84% out of 32 limit. The more clips you add and more detailed descriptions you write, you'll face the severe token limitation... This is a common problem in LLMs though.

Here's an excerpt from the API request:

{
  "model": "jev-latest",
  "questions": {
    "clip": {
      "type": "choice",
      "instructions": "Rank the available footage for a VJ preview deck using state.prompt and state.clips.",
      "criteria": {
        "Opti/Opti6.mov": "Inorganic metal square tunnel with multiple grilles. White light dots travel upward on the grilles while the camera advances at medium speed.",
        ...
      }
    }
  },
  "state": {
    "prompt": "white light metallic machine",
    "action": "candidates",
    "clips": [
      // Detailed description
      {
        "id": "Opti/Opti6.mov",
        "description": "Inorganic metal square tunnel with multiple grilles. White light dots travel upward on the grilles while the camera advances at medium speed.",
        "attributes": {
          "color.palette": ["white"],
          "material.types": ["metal"],
          "camera.translation": ["forward_into_scene"],
          "camera.translation_speed": "medium",
          ...
        },
        ...
      }
    ],
    "available_ids": [...], // Playable clips, total 111
    "candidate_ids": [...], // Choosable clips, total 109
    "excluded_recent_ids": [
      "ducky3d/Animation 6.mp4",
      "tatsuyam/aqua_10.mp4"
    ],
    "current_clip_id": "ducky3d/animation 15.mp4",
    "current_clip": { ... }, // Short description of the current clip
    "current_effects": [],
    "history": [...] // Recently played clip IDs
  }
}

To mitigate the token limit, we could structurize the queries and the clip composition. My demo only had 1ch output, but for example, we can sort clips into 2 layers; layer A for overlay clips (2D geometry, patterns, text etc) and layer B for 3D graphics, etc. On request we only send the clip list for the selected layer, will reduce the cotext 50%. We can also choose the category first and let Jev seek videos only in the corresponding sub directory (e.g. "Sci-fi > minimal techno" or "Organic > deep sea"). I haven't tested them yet though.

Conclusion

I heard this kind of auto-suggestion features are already in DJ apps, but there's no VJ apps providing this feature, AFAIK. Probably that's due to the technical difficulties in video analysis, but Jev or other AI-driven methods can help the situation.

This was just a rough prototype, but I hope people will find more interesting use cases in live audio/visual world, instead of just generating clips.

Blender as API

Fri, 18 Sep 2026 00:00:00 GMT

I heard GPT6 Astra has strong capability in 3D related tasks. I saw a tweet of my friend, showing mulitple VJ clips created by Astra with Blender, instead of let GenAIs generate video clips directly:

AstraがBlenderでVJ素材作るやつ色々研究してた https://t.co/urMPamSCj7 pic.twitter.com/1vWqflFP81
— Saina (@SainaKey) September 11, 2026

It makes more sense to me than just generating videos directly with Nano Banana etc; it gives us more control over the entire video generation workflow. Astra generates a Python script as a 3D scene setup, easy to version control. We can edit the materials on Blender, then let Astra reuse it to generate a variety of video clips with a consistent visual taste.

First output

First I started with a no-brainer prompt: "Create a cool VJ clip on Blender". Ofc the result was terrible, but at least I could confirm that the Astra + Blender workflow actually does the job.

Second I gave it some images as reference and said "Create geometric scene with metallic materials like". Astra made clips kinda okay, but it lacked the sense of color, post-processing and details. However after I threw more requests like "Add bloom, color lights and aberration" it made some progress, it was convincing to me that we can improve the output by telling what pleases our eyes betterf, just like when we do UI dev.

Then I remembered Blender 5.3 added dispersion support (rainbow-ish refaction of transparent materials like glass, water etc).

I asked him to provide me a list of ideas with test render. It was surprising that it made a variety of scenes, though some of them didn't make sense as VJ clip.

I picked some of them, added some ideas, and let it proceed to full render. Yet they are too lame to be used for VJ directly, they look just a few steps away from what I'd say it's finished. You know it's AI, the cold bloodless thing, it always try to use dumb simple shapes like square, triangle etc. In the glass flacture sample, all the fractures were strait-cut triangles. It took 20~30 iterations to make the shapes naturalistic, finess the lights and materials, adjust the post-effects.

The final outputs were like this:

To be accurate, I've seen many folks trying to control Blender with AI since GPT-3.5 era. But thanks to the fast iteration & strong compture-use ability in Astra, it's like Blender is acting like a remote renderer API, kinda more purified & abstracted than working as an MCP server.

However there's still some quirks; it can't actually judge the visual quality of the output by itself. It knows what is thought to be "cool" as a knowledge, it knows what pleases humans' eyes, but its doesn't feel pleasure. At least, not yet.

For example, I once tried to generate a particle scene. The sketche generated by GPT Image was good, but it failed to reproduce it in Blender. He understands the base shape, but can't reason how those particles should move, unlike we can easily imagine from the image. The output was just a wobbling sphere, without the flare-like flows in the sketch.

Once manage to tame Astra completely to let it generate a perfect scene, the bottleneck moves from manipulating Blender to:

Explain your demand in format AI can read
Rendering time

1. is　the problem that we're all facing in software development. But in visual development it could be even more difficult, since it's not straightforward to translate the image in our brain into text. 2. is a hard problem. It solely depends on the path tracing algorithm and the GPU power. We can use faster renderer like EEVEE or choose game engines like Unity / UE, but if you need high quality outputs, it's an necessarry expense——until AGI discovers a blazing fast pathtracing technique.

You keep rebuilidng blog systems instead of writing blog!

Wed, 16 Sep 2026 00:00:00 GMT

Yes, I know... so this is my new blog, focused on casual writing in English.

Just like softwares, any blogs can get stale. My main blog is written in Japanese. When I started posting on it, at first only my friends read it so I could write anything. But later I started writing more technical stuffs, the blog's topic got kinda narrowed, and I shifted my feeling and daily whatabout to Twitter. And I changed my job, my métier shifted to Web dev to graphics. I tried to write in the same way as before, but I felt weird friction between what I wanted to write and what the blog had grown into. And life went on, every life shift made my blog more rigid and outdated...

I wanna change this, esp given the current situation of web & socail media, I need to write my feelings more casually.

Architecture

When I was learning French last year, I used to write blogs on Hugo. But this time I wanted to make my blog more versatile, like allowing JS / iframe because I'd write technical articles with live demos.

My previous french blog

First I asked Claude for possible options for my demands. He suggested Astro, Next and other SSG frameworks, but I didn't wanna make my blog complicated... I don't wanna be bothered by npm update, I just want a static site, not a SSG with flashy new technology.

Then I let Astra (cuz I had almost unused 1mo subscription) to make up minimal blog setup with Vite. The architecture is like this:

The build script & runtime is quite short, just ~1k line of JS. It also gave me a pretty looking editor page. It has to have GitHub auth so it runs on a Cloudflare Worker, but finally no server is needed.

Editor page running on Cloudflare Worker

I'll try posting my daily stuff as often as I can, let's see...