A truism I have found useful: Models are not fungible, but they must be replaceable. To that end, I’ve spent a fair bit of time fiddling with which models are most useful for which roles in the factory. Having just shipped the 0.0.19 build, I wanted to take a moment to talk about where things currently sit.
Apparently I’m doing a series of blog posts about this project!
Part 1: Six Days Equals Six Weeks
Part 2: Specialists in the Factory
Part 3: The Invisible Work Matters
Part 4: Progress, 37 Days in
Part 5: The Meta-Game Begins
Part 6: In Which I Intervene in the Code
Part 7: Quality Requires Visibility
Part 8: Roost, Terrain, and Sound
Part 9: A Big Win, and a Change of Pace
Part 10: Scaling Process
The Roles
Having begun to adopt the approach Jeremy McEntire has come up with for Exemplar, the roles have changed slightly: Implementation is no longer a single role, or as simple as a supervisor / worker split. And the division of tasks into finer-grained chunks (e.g. implementer, PR responder) has fallen by the wayside as well. If I were still able to work on the game full-time, I’d still be looking at keeping several “lanes” of work going in a controlled manner but since my attention has become an even more scarce commodity, I’m not trying to have the factory run unattended overnight right now.
Currently there are six primary roles: Planning, Orchestration, Code Authoring, Test Authoring, Validation, and Reviewer. The Reviewer role is actually kind of two roles, however. There’s the Reviewer when operating against a plan, and the Reviewer when operating against code.
(The roles at the periphery for documentation maintenance and auditing still exist of course, but for simplicity I’ll set them aside.)
The basic flow is an interactive loop with the Planner to reach “induced understanding”. For complex plans, this is then dispatched to the Reviewer, which tackles it via multiple agent instances adopting a bunch of different personas (e.g. Red-Teamer, and Sage – along with several others). From there, the Planner launches an Orchestrator that supervises the Coder and Tester, and reports back to the Validator. Once a PR is filed, the Reviewer tackles it via a subset of its personas.
This produces a need for even more variety to avoid group think. This basically comes down to the Coder, Tester, and Validator all needing to be from different model families. Ideally the Coder/Reviewer and Planner/Reviewer use distinct families as well.
The Models
Early on, I was primarily using Anthropic models for the brunt of the work, with adversarial tasks (e.g. code review, and audits) handled by a variety of non-Anthropic models: Copilot, OpenAI’s then-current frontier model, and GLM-5.2.
In addition to this, there have been some… interesting… developments with the models themselves. In particular, Opus 5 stands out as presenting a dilemma. On the one hand, it can produce somewhat better output than Opus 4.8. On the other hand, it just cannot be made to shut the fuck up. That propensity to verbosity can also turn into aggressive over-engineering without careful oversight. Worse still, when multiple models are communicating, any of them being Opus 5 has a weird effect where the other models start talking like Opus 5. It doesn’t cause them to produce better output, it just means a fuckton more tokens getting spent and a ton more human labor in the form of reading.
Fable 5.1 seems to get the benefits of Opus 5 without the drawbacks. Of course I can only use so much of my usage alotment on it, and it burns through my alotment faster due to the higher per-token costs. Starting this month, I’ve stopped using Opus 5 and am now using primarily Opus 4.8, and Fable 5.1 for tasks where Anthropic models are appropriate.
GLM-5.2 has been set aside in favor of GLM-5.3-Flash for the most part. Almost as good as GLM-5.3 at a fraction of the cost. GLM-5.3 itself occasionally comes into the mix, and may become more prominent if it gets promoted to more supervisory tasks.
Kimi-K3 has been added to the mix.
I’ve also been playing with GPT-6-Astra. This one, I am on the fence about. The quality of output is good, but when asked to supervise other models it wound up burning a truly phenomenal amount of tokens for a relatively simple task. It may wind up in the mix as a Coder/Tester, but I am hesitant to use it for Planner, Orchestrator, or Validator at this point.
The Reviewer role is among the most token-intensive right now because of the use of multiple personas. That strongly discourages Anthropic / OpenAI model usage just to keep things economical.
So for those keeping score, the current layout is:
| Role | Model(s) | Next Steps |
|---|---|---|
| Planner | Opus or Fable | I plan to try GLM, and Kimi-K3 for this. |
| Orchestrator | Opus or Fable | I plan to try Sonnet, GLM, Kimi, and maybe a lower-end OpenAI model for this. |
| Coder | Opus, GLM, Kimi | I need to run more experiments with Kimi in this role. |
| Tester | Opis, GLM, Kimi | My next steps will depend on the results of a more exhaustive audit of the test suite. |
| Validator | Opus | I’m planning to try GLM-5.3 (both regular and Flash), as well as Kimi-K3. |
| Reviewer | GLM, Kimi | I’m largely happy with the models here at the moment, but the role itself is evolving. |
Copilot is still in the mix specifically for code reviews, of course – but I treat that as somewhat distinct from the Reviewer role since I can’t readily slot that model in and have it adopt the personas I want.
Should I wind up getting a DGX Spark, I’ll also be trying something recent from the Qwen family in various roles, of course.
Where Things Stand
The current role definitions have proven to be fairly well-crafted in terms of model substitutability overall. As noted above, GPT-6-Astra seems to be perpetually unhappy with the work product of other models, so it might need some adjusted verbiage to be viable for a Planner / Validator role. Additionally, I’m still working on the verbiage for the Reviewer personas to mitigate the unpriced false positives problem.
That said, 0.0.19 shaped up somewhat better than 0.0.18: More issues tackled, fewer bugs / misses getting landed and needing follow-up work.
A bigger picture thing is that I’m pleasantly surprised at the low rate of regressions so far in the project. Despite the superficial simplicity of the game, this project is already a fairly sprawling and expansive project involving fairly complex game design issues, all the complexities of 3D game development (including cross-platform compatibility), data analytics, and some distributed systems elements with non-trivial security considerations.
Where the numbers stand
