Stop Asking the Model What the Categories Are
Daniel Brodsky10 min read
We rebuilt issue clustering by moving the taxonomy out of clustering entirely.
Every conversation our platform ingests can produce findings — discrete, machine-detected problems. A tool timed out. The agent refused something it should have handled. A capability the user asked for doesn't exist. Findings are cheap and plentiful, and on their own they're close to useless: nobody wants to scroll forty thousand rows of "something went wrong here."
The job of issue clustering is to turn that pile into a short list of issues — the distinct underlying problems — so someone can look at fifteen things instead of forty thousand.
We've built this twice. The first version asked a language model to work out what the categories were. The second version doesn't ask anyone, because by the time clustering runs, the categories already exist. That shift turned out to matter far more than any algorithm choice.
The first approach: let the model find the categories
The original pipeline was a map-reduce with a language model at every stage.
Split the unassigned findings into small batches. Ask a model, for each batch: what distinct technical problems do these represent? It comes back with candidate issues — a title, a description, and the findings each one covers. Because the batches can't see each other, the same problem gets discovered several times under several names, so a second stage merges the duplicates into one consolidated catalog. A third stage then walks every finding and attaches it to one issue in that catalog.
When a catalog already existed from previous runs, a similarity check on issue titles tried to route findings to issues we already had rather than inventing new ones.
What "an issue" meant
This is the part worth sitting with, because everything downstream follows from it.
There was no definition of an issue anywhere in the system. No schema beyond a title and a description, no key, no rule. The definition lived entirely in a prompt, which was less a prompt than an editorial style guide:
- Group by capability or integration, not by error. All GitHub tools failing is one issue. GitHub plus Slack is two.
- Prefer root cause to symptom. "Agent says it can't do X" is a symptom; find the gap underneath.
- Split by domain. "Agent Can't Access Alert Data," never "Agent Can't Access Data."
- No generic buckets. "Various API Errors" and "Missing Tools" were explicitly banned, along with a blocklist of vague words.
- Titles read "Agent [can't / fails to / provides incorrect] [specific thing]," under ten words.
So an issue was never "a cluster of similar findings" — similarity was never measured. The model was asked to infer shared causation and give it a name. That is more ambitious than any distance metric can be: a timeout, a refusal, and a missing-tool error can be one issue if they share a cause, and no embedding will ever tell you that.
The catch is what ends up serving as identity. There was no stable key, so the title was the key. That's why the prompt spends most of its length on title formatting rules. Those rules weren't cosmetic. They were structural — an attempt to make a generated string stable enough to behave like an identifier.
What broke
Duplicates accumulated. The same problem would come back under a slightly different name and be recorded as a new, separate issue. Nothing downstream could tell that the two were the same thing, so the list grew with restatements of problems already on it.
Granularity was un-steerable. This is the failure people usually misdiagnose. It wasn't that the model was too coarse — the prompt pushed hard for specificity, banning vague words and demanding domain splits. The problem was that there was no dial. We couldn't say "give me tool-level granularity for latency problems but service-level for capability gaps." We could only rewrite prose and re-run and see what happened. It fragmented in some places and over-merged in others, and both were equally hard to correct. A better model doesn't fix this; there's nowhere to express the intent.
Wrong merges were silent. Grouping by inferred root cause is powerful when it's right. When it's wrong, you get an issue that quietly contains a timeout and a refusal, and there is no ground truth to check it against. Nothing surfaces the error. It just sits in the list looking plausible.
Nothing was explainable. "Why is this finding in this issue?" had exactly one answer: a model put it there. Not reproducible, not checkable, not something you can show a customer who disagrees with the grouping.
And underneath all of it, the thing that made every other problem worse: this was the spine. It wasn't one uncertain step in a reliable pipeline. Everything — the issue list, the counts, the trends, the lifecycle tracking built on top — rested on a foundation that was re-derived from scratch on every run and never landed in quite the same place twice. When your ground moves, you can't tell instability from signal.
The reframe: taxonomy belongs in detection, not clustering
Here's the shift, and it's a one-liner: we stopped asking clustering to discover the taxonomy, because the taxonomy now lives in what we choose to detect.
Findings aren't undifferentiated anymore. Each one is produced by a specific analyzer that already knows what it's looking for — duplicate tool calls, cacheable tool calls, deprecated models, latency anomalies, missing capabilities, abuse. That set of analyzers is the taxonomy. It's a design artifact we own, versioned in code, changed deliberately.
Once the taxonomy sits in detection, clustering has almost nothing left to decide. Findings route to a handler for their type, and each handler groups in whatever way is natural for that type:
- For most, it's a
GROUP BYon a key already present in the data — the tool name, the model name. Ten of our twelve handlers work this way. No model involved in the grouping decision at all. - For the two with no natural key — missing and broken capabilities, which are free-form descriptive text — findings are clustered by description similarity, and only then does a model write a title for the resulting group.
That's the inversion. The model went from deciding what the categories are to naming a group that already exists. Naming is a job models are reliably good at, and a bad name costs you a bad label, not a corrupted taxonomy.
GROUP BY isn't a clever simplification here. It's what's left over once the hard categorical decision has moved upstream.
What it buys
Expressiveness, first and most importantly. This is the part that surprised us. A model reading conversation transcripts could never propose "these tool calls could be batched" or "this session's token usage is anomalous" as a category — not because it isn't smart enough, but because those aren't visible in the text. They require computation over telemetry: percentile baselines, cache-hit analysis, token distributions. An analyzer computes. A summarizer only reads.
So the old approach had a hard ceiling at categories discoverable by reading a conversation, and no model improvement lifts it. Defining the taxonomy ourselves didn't just make it more stable — it made whole classes of problem expressible that were previously unreachable. Most of our highest-value detections today are things the old pipeline structurally could not have found.
Granularity became a control surface. GROUP BY tool_name is tool-level by construction. The similarity threshold on the description-clustering path is a number you can turn. Both are decisions we make explicitly rather than outcomes we observe after the fact.
Explainability stopped being binary. For most issues, "why is this finding here" is now answerable in one query: because its tool name is X. For the two fuzzy paths it's "because its description was within a set distance of the group" — still approximate, but a number with a threshold you can inspect, argue with, and tune. That's a different universe from "a model decided."
Duplicates largely went away. Issues carrying the same title as an existing one dropped from around five percent of everything created to essentially zero, while monthly issue volume went up.
What it costs
It would be dishonest to present this as free.
We gave up root-cause synthesis. The old pipeline could merge a timeout and a refusal into one issue on the grounds that they shared a cause. Nothing in the new design can do that. Grouping now happens strictly within a detection type. When the old approach got this right, it produced real insight, and we lost that.
Twelve handlers is twelve handlers. Every detection type needs one written by hand. That's real ongoing cost, and it's the price of the control surface — you can't have "exactly the granularity I want" and "no per-type code" at the same time.
Coverage gaps are now possible. If a finding arrives with a type nobody wrote a handler for, its findings simply don't get clustered. We log that loudly, because a silent coverage gap is worse than a visible one — but the failure mode exists, and it didn't before.
Where fuzziness still lives
We didn't eliminate uncertainty. We relocated it.
Two of the twelve paths still group by description similarity, because free-form capability gaps have no natural key and pretending otherwise would be worse. A model still writes every title. And one handler groups on a category the model produces as free text — a loose string doing keying work, which is exactly the pattern this post is about, still live on one low-volume path. It's the next one to go.
What changed isn't that the fuzziness went away. It's where it sits: late, contained, over a catalog that deterministic routing has already built, on paths we can point at and count. Not underneath everything.
The rule we'd write down first
If we were starting over, the rule wouldn't be about algorithms at all. It's this: decide where fuzziness is allowed to live. Keep it off the spine, out of anything your lifecycle features depend on, and confine it to one late step you can measure and rip out. That rule would have caught the first design, and it also describes what the second one actually does.
Because the takeaway isn't "determinism good, models bad." Our current system still uses a model to name every issue it creates, and still uses fuzzy similarity where no better option exists. The first design's mistake was never that it used a model to group things. It was that it let the model's output be the identity — so there was no layer underneath that stayed still.
Use a real key wherever one exists. Where none does, don't pretend otherwise — just make sure the uncertain step is small, late, and measurable, so that when it drifts you get a number instead of a mystery.
And before any of that: ask whether you're solving a grouping problem or a taxonomy problem. We spent a long time building better and better clustering for what turned out to be the wrong question. The categories were never in the data waiting to be discovered. They were a decision we hadn't made yet.