How We Made Long-Running AI Jobs Survive Worker Shutdowns

At Brizz, we run a background job that groups failures from AI agents into a short list of issues. When we tried to run a full rebuild of that list for one of our biggest customers, it failed five times in a row.

The cause was our own infrastructure. Our workers sit behind an autoscaler, and whenever it shrinks the pool, Cloud Run gives each departing worker ten seconds to finish before killing it, which is a problem when some clustering activities need up to 25 minutes of uninterrupted work. Worse, the run that looked healthiest was actually the most broken: it reported success even though it had skipped 863 of the 947 deletions it was supposed to perform.

We first looked for ways to stop workers from being killed, but eventually we changed the goal instead:

Stop trying to make worker shutdowns rare. Make them cheap.

Today the same workload runs continuously through deploys and scale-downs. Over one 44-hour window it ran 1,218 times, and although workers were shut down in the middle of 105 of those runs, none of them failed. This is the redesign that got us there.

The mismatch

Brizz watches AI agents work. Whenever an agent fails a user, we record a finding, and since there are far too many findings to inspect one by one, a background job groups them into issues.

That job runs on Temporal, which breaks work into activities, records the ones that complete, and retries the ones that fail. The activities themselves run on a Cloud Run worker pool managed by an autoscaler.

The old rebuild processed findings in large batches, where each batch was a single Temporal activity that did everything for its share of the data:

1. Group the findings into issues,
2. Generate titles for new issues with an LLM,
3. Write issues,
4. Assign findings to issues,
5. Delete old issues.

Doing all of that in one step meant a single activity could take up to 25 minutes, and that created a bad mismatch. The autoscaler was allowed to reclaim a worker at almost any time, while our unit of recoverable progress was a 25-minute activity. When a worker died after 18 minutes, Temporal could retry the activity, but those 18 minutes of work were gone.

Our retry policy made this worse, because the retries were packed too closely together. A failed activity got three attempts over roughly a minute, so all of them could land inside the same scale-down:

18m 00s   try 1   ✂ worker killed after 18 minutes of work
18m 10s   try 2   ✂ same scale-down is still happening
18m 40s   try 3   ✂ killed again
19m 10s   no tries left

Retries only help when they outlast the condition that caused the failure. Ours did not.

We also had a second, quieter failure mode. We limited each batch by the number of findings, but the slow part of the activity was the number of new clusters, because each new cluster needed an LLM call. As a result, a batch with 29 large clusters could finish comfortably while a similarly sized batch with 698 small clusters could time out.

We were limiting the wrong thing.

Why we did not just stop the kills

We considered the obvious fixes first, and each one had a catch.

Delay scale-down. Workers would be killed less often, but each kill would still throw away a large amount of work.

Give the rebuild a dedicated worker. That would avoid autoscaler kills, but deploys could still restart it.

Move the rebuild to Cloud Run Jobs. This was attractive because a job is designed to run to completion. However, it also meant introducing a second execution model, another container path, new orchestration code, and another place to debug production work.

Checkpoint inside each long activity. This would reduce lost work, but correctly resuming a partially completed clustering mutation was much more complicated than we wanted.

What these approaches have in common is that they all try to make interruption less frequent. The more useful question turned out to be the opposite one: what if interruption is normal?

Our heavy jobs are mostly long lists of independent pieces of work, each with a few expensive LLM calls. Temporal already handles that kind of workload well, as long as the workflow exposes small enough units of progress.

So instead of protecting a 25-minute activity, we removed it.

Plan once, execute in small pieces

The new workflow has three stages:

Step 1 - Plan

The plan reads the findings and makes all the expensive decisions up front: how they cluster, which clusters attach to existing issues, and which ones require new issues. Crucially, the plan writes nothing to the application database.

That property is what makes planning safe to interrupt. If the worker dies mid-plan, we lose some compute time and simply run it again, and once the activity succeeds, Temporal persists its result, so the decisions are no longer tied to any one worker's memory.

Planning up front also fixed a quality problem in the old implementation. Previously, each batch clustered only its own findings, so two findings about the same problem could land in different batches and never be compared, and making batches smaller for reliability would only have made that fragmentation worse. Now clustering happens once over the whole input, and only execution is sliced.

Persisting the plan does have one cost: Temporal limits the size of activity results. Our first version stored every finding ID in every cluster, and one large rebuild produced a 2.22 MB plan, over our 1.5 MB budget. Storing the rule that selects each cluster's findings, instead of the IDs themselves, brought the same rebuild down to 334 KB, even over a larger time range.

Step 2 - Slices

A slice takes a small part of the saved plan and carries it out:

• generate titles for new issues,
• write those issues,
• assign findings,
• remove old issues that the slice emptied.

A slice does not make clustering decisions; it only executes part of the saved plan. We therefore bound slices by the expensive operation rather than by the number of findings: each slice creates at most 20 new issues, since every new issue requires an LLM call. In production, that keeps most slices small: half finish in about a second, and 95% complete in under two minutes.

Step 3 - Finalize

Because some old issues have findings spread across several slices, an individual slice cannot always tell whether an issue is now empty. A final activity therefore runs after all the slices, inspects the actual database state, and deletes the old issues that are safe to remove.

The important change here is not really "batching" but the size of the largest indivisible piece of work. The old workflow needed roughly 25 uninterrupted minutes to make progress on one activity, whereas the new one usually needs less than two.

Idempotency is what makes small slices useful

Making activities smaller limits how much work a kill can destroy, but it does not by itself make retries safe. A worker can die after writing half a slice, in which case Temporal executes that slice again, and the second execution has to converge to the same state as the first. A few rules made that possible.

Deterministic IDs

New issue IDs are calculated from the run ID and the cluster's position in the plan:

uuid5(namespace, f"{run_id}:{cluster_index}")

Because of this, a retry generates the same IDs and overwrites the same rows. With random IDs, a retry could create duplicate issues that would then need their own cleanup logic.

Never delete first

A slice writes the new issue before moving findings to it, and only then removes old issues that became empty. That ordering means that if the worker dies halfway through, every finding still points either to its old issue or to its new one; we never intentionally pass through a state where a finding points at nothing.

Freeze values that must survive retries

Anything that could change between executions but belongs to the logical operation is fixed in the plan. Creation timestamps are a good example: our issues table is partitioned by month, so a retry that crosses a month boundary must not write the "same" issue with a different creation month.

Delete against current state

People can modify issues while a rebuild is running, so deletion cannot rely solely on what the plan believed at the start. Instead, the delete statement checks again, at execution time, that the issue is still empty and still eligible to be removed.

Working through these rules gave us our review method before shipping: for every step, ask what happens if the worker is killed exactly here? That exercise found bugs outside the main algorithm, in code we had spent much less time thinking about.

A useful progress rule

During the redesign, someone asked a deliberately hostile question:

What if workers are killed every five minutes for hours?

Answering it led to the simplest mental model we found for the problem:

A workflow can keep making progress as long as its longest retryable piece of work is shorter than the useful time available between interruptions. Each interruption costs at most one piece.

If workers disappear every five minutes, a 25-minute activity can fail forever, and retrying it more aggressively does not help. A two-minute activity, on the other hand, usually fits between interruptions, and when it does not, we lose at most those two minutes and try again elsewhere.

The rule also changed our retry policy. Slices can now retry many more times, with delays that grow from seconds to minutes, because once retries are cheap and idempotent, it is reasonable to spread them across the full duration of a transient infrastructure event.

The first production test

We shipped the redesign behind a per-run switch and retried the same rebuild that had failed five times. This time it completed with:

• 1 plan,
• 23 slices,
• 429 issues created,
• 9,362 findings assigned.

A second rebuild that evening was larger, with 34 slices, 441 new issues, and 16,904 findings processed. The autoscaler shut workers down twice during the run, yet the workflow history showed no failed run.

The failure mode had not disappeared. It had stopped being important.

Once the rebuild path proved stable, we moved the normal 20-minute clustering schedule onto the same workflow and deleted the old batching code. The scheduled path is actually the better stress test, because it runs continuously for every tenant, through every deploy and every scale-down. Here is what one 44-hour window looked like:

• Runs: 1,218
• Failed runs: 0
• Runs with a worker shutdown during a step: 105 (9%)
• Total steps: 10,396
• Steps retried: 123
• Retries caused by worker shutdown: 117
• Slice duration, p95: 106 seconds

About one run in twelve had a worker disappear during one of its steps. In each case, the step reran on another worker, repeated a minute or two of work, and the workflow completed normally.

The autoscaler still kills workers. It just makes a run a little slower now.

What we took away

• Make interruption cheap instead of only making it rare. Infrastructure eventually takes machines away, so design the work so that losing one does not matter much.
• Separate decisions from execution. Make the expensive global decisions once, persist them, and let small activities carry them out.
• Choose the unit of work around the expensive resource. We originally bounded findings, but runtime was dominated by LLM calls, so new clusters were the right quantity to bound.
• Idempotency is what turns retries into recovery. Deterministic IDs and safe mutation ordering let a partially executed slice run again without inventing cleanup work.
• Spread retries over longer than the failure condition. Three attempts in one minute are effectively one attempt when a scale-down event lasts longer than that.
• Test interruption points explicitly. Asking "what happens if the worker dies here?" was more useful than reasoning about the happy path.

The main change was not a new queue, a new autoscaler, or a new compute service; it was the unit of recoverable progress.

Once the largest meaningful step became small enough to fit comfortably between interruptions, worker shutdowns stopped being failures and became retries.