LLM alert triage

CritterWatch can ask a language model to look at a group of alerts that fired together and write a short assessment of what is probably going on. The narrative is stored against each alert it names and rendered in the console beside them.

It is off by default and must be turned on deliberately. See Cost — this calls a paid model on a schedule you do not control, and an alert storm is when it fires hardest.

What it does

Alert changes are already coalesced into one batch every ~250 ms by CritterWatch's alert accumulator — the same batch that feeds Slack and webhook notifications. Triage is a third consumer of that batch, and the batching is the point: the model is shown the alerts that fired together as one possible incident, which is the correlation an operator otherwise has to do in their head across five separate notifications.

For each batch it produces:

• a one-line summary,
• a longer assessment in operator terms,
• zero or more suggestions, each optionally naming an action the console already offers for that alert type.

What it does not do

⚠️ Inference is read-only. The model cannot take any action, and nothing it returns is executed.

A suggestion may name an action — restarting a projection, restarting a listener — and the console will render the same button it already renders for that alert type. Pressing it goes through the ordinary command path, with the ordinary RBAC capability check. A narrative that suggests something you are not permitted to do is a narrative; it is not an escalation of your permissions.

Suggested actions are also validated server-side against the remediation catalog before they are stored, so a command the model invented is discarded rather than shown. An unrecognised action renders no button — the safe direction.

⚠️ The narrative is advisory text and is labelled as such in the console. Nothing in CritterWatch reads it to decide anything: no alert is raised, resolved, elevated or suppressed because of what a model said about it.

What it looks like in the console

The narrative appears on the alert card, on the service's detail page, between the alert's own message and its remediation buttons. It is labelled "AI ASSESSMENT — SUGGESTION ONLY, NOT A DIAGNOSIS" on its face rather than in a tooltip: an operator reading a confident paragraph beside a red alert should know a model wrote it without hovering anything.

A suggested action marks the button the console already offers — it never adds one. If the model suggests restarting a projection, the existing Restart projection button gets a small marker; there is no second, AI-owned button beside it. That is deliberate and it is what makes inference is read-only structural rather than merely documented: there is no path from a narrative to an action that does not go through a button that already existed and is already RBAC-gated. It also avoids two competing button sets on one card, where the AI-suggested one would inevitably read as the more authoritative.

An alert that has never been triaged simply has no panel — which is every alert on a console with the feature off.

Setup

Two things: register a model client, and turn triage on.

The model client is yours. CritterWatch binds Microsoft.Extensions.AI's IChatClient and takes no vendor dependency, so the provider — Anthropic, OpenAI, Azure OpenAI, Ollama — is your choice, and its credentials live in your application's configuration rather than in CritterWatch's.

builder.Services.AddSingleton<IChatClient>(/* your provider's MEAI adapter */);

builder.Host.UseWolverine(opts =>
{
    opts.AddCritterWatchForSoloHost(connectionString);

    opts.AddCritterWatchAlertTriage(
        triage =>
        {
            triage.Enabled = true;
            triage.Severities = [AlertId.Severities.Critical];
            triage.MaximumAlertsPerBatch = 20;
        },
        callouts =>
        {
            // See "Cost and rate control" below. Set this.
            callouts.Budget.MaximumTokensPerWindow = 200_000;
            callouts.Budget.Window = TimeSpan.FromHours(1);
            callouts.MaximumParallelCallouts = 2;
        });
});

Options

Option	Default	What it does
Enabled	false	Off unless set.
Severities	["Critical"]	Which severities are worth a model call.
Model	null	Overrides the callout options' default model id, and is recorded on the stored narrative. ⚠️ Left null, the narrative records no model: the typed response carries no model identity, so CritterWatch would be guessing. Set it if you want that provenance.
MaximumAlertsPerBatch	20	Caps how many alerts are described to the model. The prompt still says how many were left out, so the model can say "and 340 more of the same" rather than reasoning from a silently truncated list.

Only raised and elevated alerts are triaged. Resolutions, clears, acknowledgements and snoozes are the fleet getting better or an operator acting — none is a question worth paying for.

Cost and rate control

⚠️ Set a budget. An uncapped fan-out to a paid model during an alert storm is a bill, and that is precisely the class of thing this product exists to catch.

The controls are Wolverine's, configured through the second argument above, and CritterWatch deliberately does not duplicate them — two ceilings that disagree is worse than one:

Control	Effect
Budget.MaximumTokensPerWindow / Budget.Window	A rolling token ceiling. Over budget, a callout is dead-lettered rather than sent.
Budget.MaximumPromptCharacters	Rejects an individual oversized prompt.
MaximumParallelCallouts	Concurrency cap on the callout queue.
Timeout, RetryCooldowns	Per-callout timeout and retry schedule.

Two structural properties help before any of those do:

• One call per batch, never one per alert. A hundred alerts in one 250 ms window is one request.
• Callouts are deduplicated on the identity of the alert set, so a redelivered batch does not buy a second call.

Operating it

Because a callout is an ordinary Wolverine message on the llm-callouts local queue, CritterWatch monitors it like any other message: latency, failure rate, retries and dead letters all appear on the usual screens. A model that has started timing out, or callouts piling up against a budget, look like any other unhealthy endpoint.

Dead-lettered callouts are the first place to look if narratives stop appearing — an over-budget callout lands there by design, and it is a normal, recoverable state rather than an error.

Limits worth knowing

• A narrative is a snapshot. It describes the fleet as it looked when the alerts fired, and it is not revised as the incident develops. Re-triage happens only if the alerts fire again.
• The newest narrative wins on an alert that is triaged more than once. Every one is kept in the alert's event stream, so the history of what was said is not lost.
• No narrative is produced when the model returns nothing usable, and the alert simply carries no annotation. The alert itself is unaffected — triage never gates alerting.