LLM Execution Optimization and Benchmark Discussion

[Scriptwonder]

The major pain point I am experiencing is that LLM does not have a spatial/contextual understanding of certain commands. This happens when I ask it to generate a populated city, a Dust 2 map prototype, or a garden. User prompt is meant to be simple, so I think we need to add Metaprompt and guidance for LLM, especially on Unity execution, to make it smarter.

What I envision right now is a simple GUI input/button to start, so the user can describe their intent for the further prompt / LLM can use ManageScene to gather the current JSON data and try to have an understanding of what to do. It can go more complex than that, but a starting point with LLM knowing the explicit general answer would be good.

Also, hard to find a benchmark for this kind of testing scenario! We should think of possible testing cases and generation cases to test out our tool in the future.

This can be a very long discussion in terms of optimizing LLM execution; maybe Coplay has experience tackling this issue, but I am very open to the discussion.

[dsarno]

Agreed it's a very smart idea to have a "test driven development" kind of approach, where we pick a few scenarios like you said and work on the system till it can handle them.

• Garden with simple plant shapes is a great one.
• Simple city is a good one.
• Simple start menu is a good one (lots of people asking about UI)
• 2D Galaga clone would be a more sophisticated one.

Any others?

[msanatan]

Unity's included projects are good testing grounds as well!

[Scriptwonder]

It has been a while, but I want to pick up this subject again as I am working on batch & vision input for the MCP. Is there a way we can together build a benchmark prompt that could test the MCP's robustness, with each function being tested with 2~3 tasks?

[whatevertogo]

I'm currently working on it. I've been trying to generate a tool on the MCP side for the agent to use, which can summarize the project and some dependencies (it's not yet fully functional). Instead of using a context window, it relies on memory stored in markdown files.
However, there's a challenge with AI's spatial understanding that's hard to solve; perhaps we need a better model?

I feel it's a bit complex to figure out the entire Unity project for this feature. I will try to make a preliminary function first and then hand it over to the community for improvement?

[dsarno]

This has been transferred to a discussion: #780