LLM alert triage

CritterWatch can ask a language model to look at a group of alerts that fired together and write a short assessment of what is probably going on. The narrative is stored against each alert it names and rendered in the console beside them.

It is off by default and must be turned on deliberately. See Cost — this calls a paid model on a schedule you do not control, and an alert storm is when it fires hardest.

What it does

Alert changes are already coalesced into one batch every ~250 ms by CritterWatch's alert accumulator — the same batch that feeds the console's notification channels. Triage is another consumer of that batch, and the batching is the point: the model is shown the alerts that fired together as one possible incident, which is the correlation an operator otherwise has to do in their head across five separate notifications.

For each batch it produces:

• a one-line summary,
• a longer assessment in operator terms,
• zero or more suggestions, each optionally naming an action the console already offers for that alert type.

What it does not do

⚠️ Inference is read-only. The model cannot take any action, and nothing it returns is executed.

A suggestion may name an action — restarting a projection, restarting a listener — and the console will render the same button it already renders for that alert type. Pressing it goes through the ordinary command path, with the ordinary RBAC capability check. A narrative that suggests something you are not permitted to do is a narrative; it is not an escalation of your permissions.

Suggested actions are also validated server-side against the remediation catalog before they are stored, so a command the model invented is discarded rather than shown. An unrecognised action renders no button — the safe direction.

⚠️ The narrative is advisory text and is labelled as such in the console. Nothing in CritterWatch reads it to decide anything: no alert is raised, resolved, elevated or suppressed because of what a model said about it.

What it looks like in the console

The narrative appears on the alert card, on the service's detail page, between the alert's own message and its remediation buttons. It is labelled "AI ASSESSMENT — SUGGESTION ONLY, NOT A DIAGNOSIS" on its face rather than in a tooltip: an operator reading a confident paragraph beside a red alert should know a model wrote it without hovering anything.

A suggested action marks the button the console already offers — it never adds one. If the model suggests restarting a projection, the existing Restart projection button gets a small marker; there is no second, AI-owned button beside it. That is deliberate and it is what makes inference is read-only structural rather than merely documented: there is no path from a narrative to an action that does not go through a button that already existed and is already RBAC-gated. It also avoids two competing button sets on one card, where the AI-suggested one would inevitably read as the more authoritative.

An alert that has never been triaged simply has no panel — which is every alert on a console with the feature off.

Where else it goes

Once a narrative is stored, the console also hands it to its notification destinations, off the alert path and after the alert batch that raised the alerts:

• The Slack package replies in the thread of each alert it covers that was posted to Slack. SlackConfiguration.IncludeTriage = false turns that off.
• Notification channels (Slack, Discord and Teams webhooks) get one CritterWatch AI triage post per channel, listing the alerts it covers, when the channel's severity allowlist admits the most severe of them.

Every destination carries the same "AI assessment — suggestion only, not a diagnosis" label, and suggested actions are listed by name only. No destination gets a button: the only way to act on a suggestion is the alert's own button in the console.

Explaining a dead-letter group

The same setup lets an operator ask the model about a dead-letter failure group: what the failures have in common, whether they look transient or permanent, and whether to replay, discard, fix first or investigate. It is on demand only, from the group's Explain these failures action on the Failure Groups tab, and needs dlq.triage on each of the group's services. The answer is stored with the group, labelled as a suggestion, and marks the group's existing replay or discard action rather than adding one.

It shares the alert callouts' queue, budget and parallelism cap, so the cost controls cover both. Its own settings are under DeadLetters (below); DeadLetters.Enabled = false keeps dead letters away from the model while alert triage carries on.

⚠️ Explaining a group sends data from the group's dead letters to your model provider. Message bodies are left out unless you set DeadLetters.IncludeMessageBodies, and header values are sent only for Wolverine's own routing and tracing headers; every other header is sent by name alone, with the body opt-in on or off. A dead letter keeps no stack trace, so none is sent.

Setup

Two things: register a model client, and turn triage on.

The model client is yours. CritterWatch binds Microsoft.Extensions.AI's IChatClient and takes no vendor dependency, so the provider — Anthropic, OpenAI, Azure OpenAI, Ollama — is your choice, and its credentials live in your application's configuration rather than in CritterWatch's.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddSingleton<IChatClient>(yourProvidersChatClient); // your provider's MEAI adapter

builder.Host.UseWolverine(opts =>
{
    opts.AddCritterWatchForSoloHost(dataSource);

    // builder.Configuration lets CritterWatch:Ai set the cost controls too; see "Configuration" below.
    opts.AddCritterWatchAlertTriage(
        builder.Configuration,
        triage =>
        {
            triage.Enabled = true;
            triage.Severities = [AlertId.Severities.Critical];
            triage.MaximumAlertsPerBatch = 20;
        },
        callouts =>
        {
            // See "Cost and rate control" below. Set this.
            callouts.Budget.MaximumTokensPerWindow = 200_000;
            callouts.Budget.Window = TimeSpan.FromHours(1);
            callouts.MaximumParallelCallouts = 2;
        });
});

Options

Option	Default	What it does
Enabled	false; true once AddCritterWatchAlertTriage is called	Calling AddCritterWatchAlertTriage turns triage on. CritterWatch:Ai:Enabled = false (or Enabled = false in the configure delegate) turns it back off.
Severities	["Critical"]	Which severities are worth a model call.
Model	null	Overrides the callout options' default model id, and is recorded on the stored narrative. ⚠️ Left null, the narrative records no model: the typed response carries no model identity, so CritterWatch would be guessing. Set it if you want that provenance.
MaximumAlertsPerBatch	20	Caps how many alerts are described to the model. The prompt still says how many were left out, so the model can say "and 340 more of the same" rather than reasoning from a silently truncated list.
DeadLetters.Enabled	true	Whether operators may ask for a dead-letter group explanation. Only applies when triage is on.
DeadLetters.IncludeMessageBodies	false	Sends each sampled dead letter's message body, truncated, to the model provider.
DeadLetters.MaximumSamples	5	How many of the group's dead letters are described, across every service.
DeadLetters.MaximumBodyCharacters	2,000	The longest body sent, when bodies are included.
DeadLetters.ReuseFor	24 hours	How long a stored explanation is shown again instead of asking the model, while the group is unchanged.

Only raised and elevated alerts are triaged. Resolutions, clears, acknowledgements and snoozes are the fleet getting better or an operator acting — none is a question worth paying for.

Configuration (CritterWatch:Ai)

Every option above, and the cost controls below, can also be set from configuration under CritterWatch:Ai, so an operator can change them without a code change or a redeploy:

{
  "CritterWatch": {
    "Ai": {
      "Enabled": true,
      "Severities": [ "Critical" ],
      "Model": "claude-sonnet-5-5",
      "MaximumAlertsPerBatch": 20,
      "DeadLetters": {
        "Enabled": true,
        "IncludeMessageBodies": false,
        "MaximumSamples": 5,
        "MaximumBodyCharacters": 2000,
        "ReuseFor": "1.00:00:00"
      },
      "MaximumParallelCallouts": 2,
      "Budget": {
        "MaximumTokensPerWindow": 200000,
        "Window": "01:00:00",
        "MaximumPromptCharacters": 40000
      }
    }
  }
}

• Configuration wins. It is applied after the code in AddCritterWatchAlertTriage(...), so a configured value overrides the one in code.
• An absent key changes nothing. Leave a key out and the value from code, or the default, stands. With no CritterWatch:Ai section at all, the call to AddCritterWatchAlertTriage means Enabled = true, as before.
• Severities replaces, it never adds. A configured list is the whole list. As one value it can be comma-separated, which is what an environment variable naturally carries: CritterWatch__Ai__Severities=Critical,Warning.
• ⚠️ Configuration alone never turns triage on. The host must still call AddCritterWatchAlertTriage and register an IChatClient: that call registers the callout queue and the response type triage needs. A key without the call does nothing.
• Values are read at startup, so a change takes effect on restart.
• ⚠️ Call AddCritterWatchAlertTriage once. A second call throws, because the callout queue, its budget and its defaults are one configuration in Wolverine.AI.
• ⚠️ The cost controls need the configuration passed in. MaximumParallelCallouts and the Budget:* keys are read only by the overload that takes it, AddCritterWatchAlertTriage(builder.Configuration, ...), as in the setup above. Wolverine.AI consumes them while services are being registered, and at that point no host builder has its configuration available to find, so without it those keys are not read. The triage keys (Enabled, Severities, Model, MaximumAlertsPerBatch and the DeadLetters:* keys) are read either way.

Switching triage off in an incident. An alert storm is when triage spends most. To stop it, set one key and restart; nothing else needs to change:

CritterWatch__Ai__Enabled=false

Alerts are still raised, delivered and shown as usual. Only the model call stops.

Cost and rate control

⚠️ Set a budget. An uncapped fan-out to a paid model during an alert storm is a bill, and that is precisely the class of thing this product exists to catch.

The controls are Wolverine's, configured through the second argument above, and CritterWatch deliberately does not duplicate them — two ceilings that disagree is worse than one:

Control	Effect
Budget.MaximumTokensPerWindow / Budget.Window	A rolling token ceiling. Over budget, a callout is dead-lettered rather than sent.
Budget.MaximumPromptCharacters	Rejects an individual oversized prompt.
MaximumParallelCallouts	Concurrency cap on the callout queue.
Timeout, RetryCooldowns	Per-callout timeout and retry schedule.
DeduplicateCallouts	Claims each callout's deduplication id before calling the model. Off by default, and not available on a standalone console; see below.

⚠️ There is no budget unless you set one. Wolverine's defaults are MaximumTokensPerWindow unset (no ceiling), a one-minute Window, five MaximumParallelCallouts, a two-minute Timeout, and retries after 1, 5 and 15 seconds. A prompt the model cannot answer in the requested shape, and a callout over budget, go straight to the dead-letter queue rather than being retried.

Two structural properties help before any of those do:

• One call per batch, never one per alert. A hundred alerts in one 250 ms window is one request.

• Callouts carry a deduplication id derived from the identity of the alert set, so the same alerts fired together always produce the same id. ⚠️ Wolverine only acts on it when you opt in, with callouts.DeduplicateCallouts = true and Wolverine's message deduplication (opts.Durability.MessageDeduplicationMode) on a store that supports it. Without both, a redelivered batch can buy a second call.

  ⚠️ A standalone console cannot opt in. It runs Wolverine's TypeLoadMode.Static against its own pre-generated code, and under Static mode Wolverine.AI runs callouts on a pre-built handler that has no deduplication step, so since Wolverine 6.59 DeduplicateCallouts = true stops the console at startup with an InvalidOperationException saying so. The budget is the control that applies there.

Operating it

Because a callout is an ordinary Wolverine message on the llm-callouts local queue, CritterWatch monitors it like any other message: latency, failure rate, retries and dead letters all appear on the usual screens. A model that has started timing out, or callouts piling up against a budget, look like any other unhealthy endpoint.

Dead-lettered callouts are the first place to look if narratives stop appearing — an over-budget callout lands there by design, and it is a normal, recoverable state rather than an error.

Limits worth knowing

• A narrative is a snapshot. It describes the fleet as it looked when the alerts fired, and it is not revised as the incident develops. Re-triage happens only if the alerts fire again.
• The newest narrative wins on an alert that is triaged more than once. Every one is kept in the alert's event stream, so the history of what was said is not lost.
• No narrative is produced when the model returns nothing usable, and the alert simply carries no annotation. The alert itself is unaffected — triage never gates alerting.