The Granularity Problem in Issue Clustering

How we think about turning thousands of AI-assistant failure reports into a dashboard that ten different VPs can all agree with.

Two VPs walk into a dashboard

Picture an AI assistant embedded in a large SaaS product. It answers questions, creates tasks, builds automations. Sometimes a user asks for something it can't do — "invite the AI notetaker to my meeting," "send this as an email," "build me a chart." Our system detects each of these moments and records it as a missing-capability issue.

Now two VPs open the dashboard.

The first one scrolls for two seconds and says: "This is way too granular. I don't need forty rows. Aggregate it for me."

The second one looks at the aggregated view and says: "External integrations" — that's useless. Which integrations? Break it down.

They're both right. And here's the uncomfortable part: most analytics systems can't satisfy both, because the grouping was decided once, at ingest time, by a clustering algorithm with a fixed threshold. Whatever granularity fell out of that threshold is the granularity everyone gets. One persona wins; everyone else files a feature request.

This post is about doing it properly. It goes in three passes: what the product actually needs (philosophy), why the obvious ML approach fails in an interesting way (the fun part), and what to build instead (the practical part).

The data that motivates everything

A real ten-day window from one production assistant, to keep us honest:

511 distinct missing-capability issues, detected from ~6,200 sessions.
About 7% of sessions and 11% of users hit at least one gap.
The top 10 issues cover only 15% of affected sessions. 91% of issues were seen in one session or fewer.

That last bullet is the whole game. There is no "fix these three things" answer hiding in this data. The demand is a long, fragmented tail, and the only way to reason about it is to group it. Which means the grouping is the product. Get it wrong and the dashboard is either a 511-row spreadsheet or a useless pie chart with one big slice called "Other."

Pass 1: What the product actually needs

Before any ML, write down what "good" looks like. We landed on seven principles:

Drill-down is the primary gesture. Nobody should choose a global setting to peek one level deeper. Every group expands in place.
Default depth should be demand-weighted, not uniform. If "External integrations" holds 31% of all demand, show its sub-groups by default. If "Undo & revert" holds three sessions, show one row. A tree cut at uniform depth is what makes dashboards feel simultaneously too granular and too mushy.
One coarseness control. A single "fewer groups ⟷ more groups" slider serves the "aggregate it for me" persona in one click, with zero taxonomy knowledge.
Remember each user's choice. Granularity preference is a stable personality trait. Asking people to re-derive it every visit is the real failure mode.
An agent as the escape hatch. No fixed hierarchy survives contact with every persona. "Break this down by vendor" should be a sentence you type, not a roadmap item you wait for.
Numbers must reconcile across zoom levels. When the 10-group VP and the 40-group VP argue in a meeting, their totals have to match. The first time they don't, the dashboard is dead.
Groups must keep their identity over time. A trend line that breaks every time the system re-clusters is worse than no trend line.

Notice that principles 1–4 all assume something structural: that the groups form a hierarchy you can slice at different depths. So the obvious move is to make the clustering hierarchical and let users pick the cut. Which brings us to the trap.

Pass 2: The trap — dendrograms are geometry, not meaning

Our issues are deduplicated by embedding their text and running agglomerative clustering. Agglomerative clustering doesn't just give you clusters — it gives you a full merge tree (a dendrogram): every issue starts alone, and pairs of clusters merge, closest first, until everything is one blob. Cut the tree low, you get many small groups; cut it high, you get few big ones.

So here's the seductive idea: persist the dendrogram, and make the granularity slider move the cut height. Ten groups for the first VP, forty for the second, same tree, everyone happy. We almost shipped this. It's wrong, and it's worth understanding exactly why it's wrong, because the failure is subtle.

The question that kills it

Why would the embedding cluster Zoom and Slack together? Maybe it puts them in a 'chat systems' cluster instead. It's undefined.

Exactly. Take three issues:

"Unable to send Slack direct messages"
"Unable to access Zoom chat messages"
"Unable to send WhatsApp messages"

What's the parent category? Depends which axis you organize by:

Vendor — Slack things / Zoom things / WhatsApp things
Action — sending things vs. reading things
Object — all three are about messages
Product boundary — all three are third-party integrations

Every one of these axes is latent in the text simultaneously. The embedding space superimposes them all, and when the clustering algorithm merges clusters, it follows whichever axis happens to dominate the local distances. "Send Slack DM" might merge with "send WhatsApp message" first (action wins) or with "read protected Slack channel" first (vendor wins). Both are geometrically defensible. Neither was chosen by anyone. In ML terms, the intermediate structure is non-identifiable: many qualitatively different hierarchies are equally consistent with the same pairwise distances.

And the category the VP actually wanted — "third-party integrations" — is worse than ambiguous. It's a statement about your product's boundary: which systems are yours and which are someone else's. That fact appears nowhere in the text of the issues. No similarity metric can recover information that isn't in its inputs. Expecting the dendrogram to produce that node is expecting the algorithm to know your business.

There's a second, quieter failure: instability. Dendrogram internals are exquisitely sensitive to the corpus. Add a week of new issues, re-cluster, and the internal nodes reshuffle — different merges, different "categories," no persistent identity. There goes principle 7, and every trend chart with it.

The reframe: two problems wearing one trenchcoat

The fix starts with noticing we've been using one tool for two fundamentally different jobs:

Job A — "Are these two reports the same issue?" This is entity resolution. It has objective ground truth: two users complaining they can't send emails are describing the same gap, and a human labeler will agree. Embeddings plus agglomerative clustering are genuinely excellent here. Keep them — but only up to the leaf level, tuned for precision (a wrongly merged leaf poisons everything above it; an under-merged one just adds a tail row, which the next layer absorbs).

Job B — "What kind of issue is this?" This is categorization against an ontology. It has no ground truth in the data — "third-party integration" exists because someone decided the product's boundary, not because the text said so. Categories must be defined, then issues classified into them. The moment you see it this way, the tooling becomes obvious: classification is something we're very good at, especially now.

The one-sentence version: similarity is discovered; meaning is declared. Dendrograms discover. Taxonomies declare.

Pass 3: What to build instead

Five pieces, in dependency order.

1. Leaves: keep the embedding dedup

Findings (per-session events) cluster into leaf issues exactly as today. Evaluate it honestly as entity resolution: maintain a small labeled set of pairs, report pairwise precision/recall, tune the threshold for precision.

For each leaf, run an LLM extraction into a small fixed schema:

vendor:  slack | zoom | gmail | whatsapp | … | null
action:  read | create | send | automate | export | configure | undo
object:  message | meeting | automation | dashboard | file | …
scope:   internal | external          ← "is this our product or someone else's?"
hint:    capability_gap | defect | security | out_of_scope

Why facets and not just categories? Three reasons.

First, facets are falsifiable. "Does this issue mention Slack?" is checkable; "is this cluster coherent?" is vibes. You can build a 100-example golden set and measure per-field accuracy. That's what "more formal and robust" means in practice — not fancier math, but claims you can actually test.

Second, facets dissolve the multi-axis problem instead of fighting it. Remember that different personas want different axes, not just different zoom? Every axis is now a GROUP BY: group by vendor for the partnerships VP, by action for the UX lead, by scope for the platform owner. One extraction pass, many trees — and scope: external is precisely the "third-party integration" fact that embeddings could never recover. Now it's a field, because we asked for it.

Third, the hint field mechanizes triage. In our real dataset, two of the seventeen high-severity "missing capabilities" were actually a defect (the assistant failing at its core job) and a security issue wearing the wrong label. A facet flags them; a treemap never will.

3. Taxonomy: a declared, versioned artifact

The default hierarchy — the one most users see — becomes an explicit object, not an emergent one. Each node has a stable ID, a name, a definition written in prose, a parent, and lineage metadata:

External integrations
  "Gaps that require reaching systems outside the host platform."
  ├── Communication (Slack, email, WhatsApp…)
  ├── Meetings & calendar (notetaker, scheduling…)
  └── Web & files (search, URLs, external documents…)

Assigning a leaf to a node is classification: give an LLM the taxonomy (names and definitions) and the issue, get back a node or ABSTAIN. A cheaper variant that handles the easy 80%: embed the definition texts as anchors and assign by similarity. Note the sleight of hand that makes this legitimate where dendrogram-cutting wasn't — we're no longer measuring similarity between undifferentiated peers and praying the geometry means something; we're measuring similarity to a concept someone defined. Same embeddings, completely different epistemic status.

ABSTAIN goes to an unmapped pool, which is rendered in the UI, prominently. It's the taxonomy's inbox. Hiding it would silently censor exactly the novel demand you most want to see.

4. Evolution: change by proposal, not by reshuffle

When the unmapped pool grows past a threshold (say, more than 5% of weekly affected sessions), a job clusters only the unmapped leaves, drafts candidate node definitions, and emits a taxonomy diff — "add node Whiteboard operations under Content; here are 14 issues that would land there" — for approval, by a human or an auto-approve policy.

This single design choice buys back stability. Node IDs persist across versions; splits and merges record lineage; historical rollups stay computable under any version by mapping through that lineage. The taxonomy changes the way schemas change — by explicit, reviewable diffs — instead of the way clusterings change, which is "everything, slightly, all the time."

5. Rendering: cut by demand, aggregate from leaves

Granularity finally becomes what it should have been all along: a view-time parameter. Given the tree and a window, expand a node if it carries more than α of the session mass; collapse the rest; merge crumbs into "Other." The coarseness slider just moves α. Ten groups and forty groups are two cuts of one tree, deterministic and instant.

And the reconciliation rule, which sounds like a footnote but is load-bearing: every node metric is computed from the underlying leaf-level events at query time — never by summing children. Unique users overlap across child nodes; sum them and your zoom levels will disagree with each other by a little bit, someone will notice, and (see principle 6) trust dies. Distinct-counting from the leaves makes every persona's view sum to the same totals by construction.

The agent slots in here too, almost for free: "break integrations down by vendor" is just the same query endpoint with different parameters, rendered inline, with "pin this view" turning a good answer into that user's saved default. The agent never invents groupings on its own — it parameterizes the same layer everyone else uses, so its numbers reconcile too.

The takeaways

If you remember four things:

Granularity is a property of the viewer, not the data. Any system that fixes it at ingest time has chosen one persona and disappointed the rest. Make it a view-time cut.
Similarity is discovered; meaning is declared. Embeddings answer "are these the same?" — keep them there. Categories like "third-party integration" encode business decisions that aren't in the text; they must be defined, versioned, and classified into, not hoped for.
Prefer falsifiable structure. Facet extraction and definition-anchored classification can be measured against golden sets. "The clusters look reasonable" cannot. When someone asks why Zoom is under integrations, the answer should be a definition you can point at — not "the embedding space said so."
Stability is a feature you design, not a property you get. Persistent node identity, lineage through splits and merges, and change-by-reviewable-diff are what keep a trend line meaningful across months.

The deepest shift is the small one in the middle of this post: realizing that "cluster the issues" was always two jobs in a trenchcoat. Pull them apart, give each the tool it deserves, and the two VPs from the opening can finally look at the same data, at different altitudes, along different axes — and agree on the numbers.

The Granularity Problem, or: Why Your Embeddings Don't Know What a "Third-Party Integration" Is