Rohan Mayya

A new kind of primitive

I’ve been trying video editing with Hyperframes and Remotion. They are both phenomenal, without me having to touch a video editor.

I use Unreal Engine in my day to day work. The primitives here don’t lend itself well to LLMs. There’s a reason why the UE MCP doesn’t work well - models don’t have the option to write shaders, programmable rigs or particle systems in code/text. You’ll see the exact opposite in ThreeJS however, which explains the 100s of functional demos on X.

For a long time I thought this could be solved by brute forcing better computer use, but I’ve changed my mind.

The reason is cost and speed.

Even in a perfect world where a model can achieve more, it’s going to take forever (and burn a lot of tokens in the process) to work with these platforms. So much so, that better future LLMs will likely create its own abstractions to interact with these tools directly.

Progress will be gradual, because tools/platforms will first want to think through and build DSLs or APIs that models can use to interact with them freely.

If you see Remotion for example, you can consistently make fast, precise edits without going into an editor. However, any routine Control Rig operation, even if simple in Unreal Engine, takes at least 6 or 7 minutes, and is often slow and wrong. (basically unusable). It’s the same with something like Premiere Pro or DaVinci Resolve.

Now you may ask me, isn’t this net bad? You’d still want to make precise edits when you need them, that sometimes is not possible with an LLM. That’s where I think the best approach is hybrid.

The best reference I have is paper.design.

Paper is a Figma alternative but your agent has free reign over everything in the editor. Plus, you can make precise edits in editor just like you can in Figma. I’m quite sure their primitives are different (i.e well thought out for our era of software). This is where it’s all going, and one day you’ll see a video editor built on top of Remotion/Hyperframes or something similar but gives you precise control like Premiere Pro.

Every tool will need a lossless textual representation. A few examples:

Non-textual representations also bear worse training data, which means worse priors for a model.

Bear with me, because all of the above may sound painfully obvious to you, but you’ll be surprised how much friction there is in non web development software.

All this to say, many industries (game development, film, 3d, CAD etc) do not have good primitives. Maybe it’s time to rethink the game engine after all.