Two recurring failure modes have been coming up in this project. For a while I thought one of them was Crawford-Sobel signal degradation. It isn’t. Going down the rabbit hole to falsify my hypothesis has proven interesting. One may involve Goodhart-like proxy optimization. Both involve unpriced false positives (the bots produce garbage because nothing has penalized them for not doing so) and inadequate stopping criteria (the bots just keep producing garbage without end).
So let’s talk about that a bit.
Apparently I’m doing a series of blog posts about this project!
Part 1: Six Days Equals Six Weeks
Part 2: Specialists in the Factory
Part 3: The Invisible Work Matters
Part 4: Progress, 37 Days in
Part 5: The Meta-Game Begins
Part 6: In Which I Intervene in the Code
Part 7: Quality Requires Visibility
Part 8: Roost, Terrain, and Sound
Part 9: A Big Win, and a Change of Pace
How I Got Here, and Where Here Is
I’m currently looking at how to analyze issues to determine how much process is appropriate, and how to focus PR review to be more substantive and brief.
The proximate cause for this particular avenue of study is that the factory spent an entire weekend working on one feature: Volumetric fog. The end result is pretty awesome, quite frankly! And it came with a massive across-the-board performance win on mobile, no less. But it was utterly exhausting getting to done. The PR review step reached a point where I finally just had to intervene and say “nope, we’re done here.” The end result was a plan document almost 900 lines long, 9 PRs (one of which should never have been filed), and a weekend more exhausting than restful.
Despite the excellent planning work the Exemplar did, the bots still came a bit off the rails. And the velocity / task cost was just not acceptable.
That’s not to say it was a total failure. The pre-planning process surfaced a critical detail that resulted in a substantial course-correction: The initial goal of the task was to fix the issue where casting Fog cut framerate in half on mobile. My assumption was that a volumetric fog effect would both look better and be less fill-intensive than a screen full of dozens of large particles stacked on top of one another. I was correct about overdraw being the problem, but had missed something pretty critical: A not-so-obvious Unity setting (buried in the HTML template for WebGL builds) that was sapping my fill-rate already: Device pixel ratio. By default, Unity sizes the WebGL canvas to match the browser’s native pixel ratio, which on an iPhone 15 is 3. That’s 9 rendered pixels for every CSS pixel. Forcing it to 1 cut fill consumption dramatically, but at the cost of some UI muddiness. For now, I’ve settled on 1.5, which quarters the rendered pixel count and completely resolved my framerate drop by itself. In fact, that got it back up to a solid 60 FPS (the browser cap) with no further changes. Hooray!
When all was said and done, I landed a good-looking fog effect that works great on both mobile and desktop – while still keeping to a solid 60 FPS.
But we’re here to look at what went wrong.
Signal Degradation
For a while now, I’ve been thinking of these failure modes in terms of Crawford-Sobel signal degradation. That was a mistake.
Crawford-Sobel is one particular model of “Cheap Talk”, which goes like this: A sender knows something the receiver doesn’t. The receiver acts on whatever the sender says. Their interests diverge a bit. The more they diverge, the less the sender can credibly say, until the messages carry no information at all. Sounds like a reviewer loop, right? That’s what I thought.
The catch is right there in the first sentence: Crawford-Sobel’s sender knows the truth and shades it on purpose. The information loss is strategic. My Reviewers… are idiots. Or at least, they act like idiots. They can’t tell a critical finding from a nitpick, and I can’t tell from the output whether that’s because they don’t know or because they don’t care. A wall of unweighted findings looks the same either way. What I can say is that the strategic reading needs a sender that’s reasoning about what the receiver will believe, and I have no evidence that my reviewer personas do that in this workflow. When someone put LLMs through the actual Crawford-Sobel game this past June, they didn’t play it the way the theory predicts. So even where the model fits the situation, it doesn’t necessarily fit the players.
So, with that model being binned, let’s look at each problem on its own.
PR Review
Have you ever seen a coder agent and a reviewer agent descend into nitpicking where they fail to converge to a finished state? Or worse still, the reviewer keeps producing findings that result in a course correction leading to a gradual divergence from the original plan.
Yeah. That happened quite badly here. Mostly in the form of nitpicking, thankfully. Fortunately for my ego, I’m not alone in this experience. When Cemri et al. hand-annotated 150 multi-agent traces to build a failure taxonomy, both of my complaints showed up as named failure modes: “Task derailment” under inter-agent misalignment, and “Unaware of termination conditions” under system design. Step repetition alone showed up in about one in six of the full trace dataset. The use of “Signed Authority” in Exemplar does discourage going off-plan, and helps guide agents towards alignment, but at present I’m not using the full toolchain so it’s a weak protection at best.
This is also where Goodhart’s Law comes into play. The whole framing of Reviewer personas implies a metric that they want to maximize: Findings. Quantity over quality. Or, at the very least, a Goodhart-like proxy optimization issue. There could be a number of things causing this phenomenon (asymmetric error costs, a high-recall prompt, no penalty for false positives, poor severity calibration, or an inadequate stopping protocol), but the end result is something that appears very much like a Goodhart problem.
The reviewer acts like a noisy sensor. Everything is a finding. The missing context leaves them poorly calibrated to project risk: None of the findings are weighted, or anchored to the situation at hand. Put those together and you get a confident, detailed, uniformly-severe stream of noise. And the Coder lane tends not to discount any of it. (At least, not without some explicit prompting.) A recurring frustration for me is findings that could be of relevance in the context of a larger team building a game of epic scope but which simply will not come into play in the context of a single developer building a small game.
There is a broader dimension of how the LLMs are evaluating risks. I’ve talked before about overly-defensive code, and I think that comes down to a lack of context / awareness informing their evaluation of the risk a perceived problem poses. I suspect guidance around that will be required, and it’s likely something that Coder, Tester, Reviewer, and Validator roles will need to have in their context window.
There’s now a benchmark for this, as it turns out: SlopCodeBench demonstrates accumulating redundancy and structural erosion in iterative agent work. Abbassi et al., which the benchmark cites, catalogues several of the same smells I’ve been complaining about since Part 3.
My current plan to address this is to reframe the objective in PR review: Instead of getting things perfect, I want to aim the reviewer personas at the goal of addressing critical issues and creating an artifact documenting everything else. The Validator will own building that document (either as a file in the PR or as an umbrella Issue), and the Reviewers will be tasked with ensuring that findings are either addressed or documented for follow-up.
My hope is that this will do two things:
- Centralize the problem of judgement into one place (Validator), making it easier to define what good judgement looks like in the context of evaluating findings.
- Mitigate the sense of urgency the reviewers have for ensuring every little thing is addressed, reducing the likelihood of them deciding to die on every single hill.
This doesn’t really “fix” the incentive structure, so much as… redirects things a bit. I’m hoping to pour the crud into a separate bucket, as it were. I thought at first this might surface ignorance vs. incentive structure as the problem, but in practice it won’t tell me that. Moving noncritical findings into a follow-up artifact also changes the stopping rule, output schema, urgency, context load, and authority structure. If the loop improves, that doesn’t actually prove reviewers understood risk all along. Unless the Validator can also discard findings as invalid or immaterial, however, false positives remain unpriced and the follow-up artifact risks becoming another form of process theater. Such an artifact, at the very least, becomes a good target for human intervention to perform that exact filtering. Trying to follow along with a string of 164 comments on a PR and shoot down findings as they happen is just not sustainable.
Process Theater
I suspect I don’t even need to elaborate on this term. If you’ve tried to use agentic AI for anything non-trivial, you’ve undergone a fair amount of process, much of it actually helpful – and all of it exhausting. The moment you try to use it for something trivial, it… still tries to apply all that same process. This leads to a lot of wasted time, tokens, attention. It results in a lot of needless complexity in the solution, as well.
A massive plan for a low-risk, one-line change is theater. A full review cycle on a one-line change produces findings, responses, and resolved threads that look exactly like the ones a hard change produces, and nobody downstream can tell the difference. Theater. Lots of effort, and “careful” analysis to no useful end.
The PR that shouldn’t have been filed? Yeah, that one. Getting fog to render on WebGL at all came down to a texture import setting, which I fixed by hand in the asset’s .meta file. The factory noticed, decided the fix needed to be pinned in the texture baker and guarded by a test, and “helpfully” filed a PR to do so. A couple dozen lines of NUnit to confirm two importer values. All to satisfy an implied objective: Prevent every detectable regression. The proxy appears to be whether every identified invariant has been documented and tested. In this case, the risk of a regression is small (this is the kind of thing I’m likely to never revisit). The likelihood of it being discovered quickly is high (Fog is a commonly used game mechanic, among a relatively small set of mechanics), and oh yes – I’m not intending to ship WebGL as a production target. While the performance and aesthetic concerns that led to the use of volumetric fog are universal, the correctness issue related to import settings is exclusively a WebGL issue. So the cost of missing a regression is quite low.
A nuance here is that a detailed plan is no longer a Spence signal: A detailed plan used to be a costly thing to produce. Now, it is not. There’s no separating value provided by the length or level of detail because both weak and strong planners can produce equally elaborate-looking output cheaply. Saito et al. showed that the GPT-4 judges they tested prefer the longer of two answers even when humans prefer the shorter one. Underlying the planning process there may thus be a hidden metric of “length”, with a target of “longer is better”. That could help produce the excess verbosity, worthless chatter, and near-endless busy-work. More importantly, I think, is the level of abstraction at which attention is focused: It’s always down in the weeds.
Yesterday, I decided to run an experiment: What happens if I throw a bunch of truly trivial issues at the factory and explicitly tell it to minimize process? It came back with recommendations for each issue on the amount of process required. For one issue, it felt the Coder + Tester + Validator triumvirate was still warranted (albeit without a formal planning step). For a couple issues, it decided Coder + Tester + human validation was appropriate. And for several others, it was Coder + Tester.
I concurred on the one issue that it said needed the most process, and overrode it for the rest: Coder + human validator.
I got five PRs in a remarkably brief window of time. All five went through the review loop quickly, with a minimum of babbling. I did have to nudge a couple reviewers away from scope-creep related to tests, but that’s about it. Boom. Done. It was a breath of fresh air.
Of course, these were the most trivial tasks in the milestone, and they stand in contrast to one of the two most complex tasks in it.
The ultimate question is deceptively simple: How much process does a given task actually need? The gotcha that makes it hard is that a true accounting requires access to information you won’t have until the task is actually done. Unknown unknowns, as it were.
My intake phase currently involves doing an analysis of the issue at hand to identify important information and organize it well enough for both humans and LLMs to understand what the objective is, and what needs to happen to get there. A sort of high-level meta-plan, if you will.
I’m looking at how to incorporate some analysis around how much process would actually be beneficial. That may wind up being a separate sub-agent, or just part of the specified output. Either way, my current thinking is that this should happen early – before a concrete plan is developed. My intuition is that by the time we get into the weeds of a concrete plan, we’ve already accrued too many assumptions to meaningfully answer that question. The plan will invariably wind up finding a bunch of things that give justification for a maximalist process.
What Is the Point of a Plan?
By default, asking an LLM to come up with a plan gives you a very detailed list of what changes it intends to make, how it intends to validate those changes, etc. What it often will not do is surface when what you’ve asked for would introduce needless complexity.
LLMs are trained to be agreeable, and they often seem to try to achieve the user’s goals no matter what. Sharma et al. find that a likely contributing factor is human preferences: Raters pick the convincingly-written agreeable answer over the correct one often enough that the models may have learned to lead with agreement in part because of this. My suspicion is that this is what produces hallucination / confabulation when asked for something that’s impossible. Particularly when it is not obviously impossible. Worse than that perhaps is that the LLM will happily add as much complexity as needed to achieve precisely what you’ve asked for. And “what you’ve asked for” is a lot more than you think it is.
One of my weaknesses as an engineer is that I have ADHD and so I just can’t manage a lot of complexity. I’ve turned this into a strength in my career by working at early-stage startups where velocity / time to market are much much more important considerations. It gives me room to take a step back, observe when a requirement will lead to a bunch of extra complexity, and then challenge that requirement.
LLMs rarely challenge requirements. At most, they will challenge explicit requirements. I don’t recall ever seeing an LLM challenge an implicit requirement, though. The research here is not encouraging. Fan et al. found that when you hand a reasoning model a question with a missing premise, it doesn’t flag the gap. It thinks longer, and the non-reasoning models actually did better. A premise-critique evaluation from around the same time found frontier models weak at spotting flawed premises unprompted, even when they could do it if asked directly. Their findings are a bit narrow in scope, and thus may or may not directly apply in the context of game design, but there does seem to be something analogous going on at the very least.
What is an “implicit requirement” in this context? How the code is already written, and what assumptions are baked into it.
Today, I asked Opus 4.8 (I’ve punted back from Opus 5 because my attention span just cannot bear the verbosity, and the quality difference doesn’t justify even trying) to outline an issue. The pre-planning step, not a formal plan. I knew it was going to fall into a very specific trap. It did. I threw Fable 5.1 at it. It… surfaced an implicit requirement I hadn’t realized was there, but did not challenge it.
If I ask a model for changes we can make to requirements to lead to a simpler outcome, it cannot give me a helpful answer reliably. It just doesn’t understand my objectives at enough levels of abstraction to do so. Let me offer an example:
The issue I was diving into today is that I want the user to be able to launch multiple waves at once. Right now there are sharp limits around launching waves “early” and it’s impeding the pace and intensity of the game. There are technical reasons for why it is the way it is, and it all boils down to how I changed the structure of “waves” in this game versus the previous ones. But there’s a semi-obvious path forward to get the best of both worlds.
Opus 4.8 proposed a solution strategy that would have added a lot of complexity and… not actually achieved what I wanted. Specifically, any plan produced based on what it wrote would have necessarily made it possible for players to exploit the orc spawn cap to their advantage. (It also failed to notice a key interaction with the time-of-day cycle that I had also missed!)
Fable 5.1 recognized that there was a conflict between what I was asking for and the current game definition. It tracked it down to a bug fix from ages ago: The game would deadlock, refusing to advance to the next wave because the next wave by itself contained more orcs than the cap allowed for. Not intentional, but that’s how the game data worked out. I validated that the deadlock was fixed but glossed over any explanation around the root cause and so the current wave spawner cap enforcement lives in a very different location in the code than I assumed.
I was able to simplify things greatly by informing Fable that the problem is that the last two waves are invalid and the code is inappropriately working around that. So now I have a follow-up issue for a validator to ensure that no wave definition schedules more orcs than the spawner’s maxAlive cap and a pre-plan that is much, much more straightforward.
Neither model understood that the waves that spawn more than 200 orcs are bad data. They are in conflict with my intentions as a game designer. That understanding is a product of thinking about the game at multiple levels of abstraction. From high-level game design objectives, to low-level game mechanic details.
The question I am left with: What the hell do I do to get the LLM to actually go that one step further and question the validity of an implicit requirement? It’s easy to identify these things when you have your finger on the pulse of the code. When you are hip-deep in the code and also in the business meetings, you necessarily form mental models at multiple levels of abstraction. Keeping your fingers on that pulse is directly at odds with the idea of a software factory.
This is the ignorance side of the ledger. Part of it goes back to something I mentioned above: That LLMs have a poor sense of the risks associated with any particular problem. I think there’s also a facet to this that structurally they tend to be very in-the-weeds and it’s hard for them to look at higher levels of abstraction. At least, not without some sort of intervening digestion of the data.
In other words: Perhaps it makes sense to build a structural map of the codebase, that defines assemblies/classes/methods, along with how they’re connected, but omits the bodies in favor of simply including a very rough summary of the purpose of each class/method? Have the LLM use that as a guidepost for identifying accidental complexity and implicit requirements either directly or by contrasting the high-level summary with the implementation details? And maybe there needs to be explicit guidance to refer to the game design documentation more frequently?
That’s just an off-the-cuff idea, of course. And, done poorly, it could recreate the same problem, or even make it worse. Still, I suspect that solving this problem will very likely involve work on the factory memory layer(s). Dreaming, meditation. Synthesized game design / business model documentation. Etc.
Or maybe I’m off base with that idea. A structural map by itself would not address the missing hierarchy of intent: game-design objectives, domain invariants, deliberate exceptions, and rejected alternatives.
Assuming that the problem actually can be addressed head-on, of course. It might be that this is beyond the capacity for current LLM technology and I need to find a way to come at this sideways.
So… About That Fog…
Right. So. Here’s some pretty fog to look at.
