Skip to content

Release Notes ​

What is new and what was fixed in each release of CritterWatch, newest first.

If you are moving a running console between versions, read Upgrade Notes as well — it carries the changes that affect a deployed console's database schema, stored data, or monitored-host compatibility, and the manual steps for the ones CritterWatch cannot handle on its own. This page tells you what changed; that page tells you what you have to do about it.

1.4.0 ​

Released 16 November 2026 — handler percentiles, shards the console has stopped hearing from kept out of lag, a push splitter that respects the console's decode cap, configurable Prometheus label keys, and fixes to composite rollouts, agent counts and service health counts.

1.4.0 is cut from the 1.x branch: it stays on Wolverine 6.64 and net9.0;net10.0. main is the 2.x line (Wolverine 7). Read Upgrade Notes before upgrading.

New ​

  • Handler latency percentiles. The service page's Metrics tab shows p50, p95 and p99 execution time per message type over the last hour, from Wolverine's wolverine-execution-time histogram, read through a bound Prometheus or VictoriaMetrics source. A type with no executions in the window reads not reported, never 0. A value past the histogram's top bucket reads above top bucket. A source that keeps no histogram (Datadog, App Insights, the console's own store) says so. New read: GET /api/critterwatch/services/{name}/handler-percentiles. (#1566)
  • A shard the console has not heard from in 30 minutes stays out of Max Lag and the lag verdict. Each progression row now records when a report last named it, changed or not, and the shard grid shows not observed since … for a row past 30 minutes. Before this, a row last reported weeks ago could set a service's worst lag. A row written before 1.4 counts as unknown, never stale, until its first report. ⚠️ Services keep restating quiet shards on their full pass (FullSnapshotInterval, 2 minutes by default); a service configured with a full pass longer than about 25 minutes will see its quiet shards read not observed. (#1658)
  • Configurable message-type and exception-type label keys on the Prometheus and VictoriaMetrics sources, for a backend that renames Wolverine's tags (Dash0 stores message.type as rpc_message_type): MessageTypeLabelKey, MessageDestinationLabelKey and ExceptionTypeLabelKey, defaulting to Wolverine's spellings. They are used to group and to read names back, in the scrape, the dead-letter breakdown and the handler percentiles. A scrape whose series carry no configured message-type label logs one warning naming the key. See When your pipeline renames Wolverine's labels. (#1670)
  • A Dynamic Consistency Boundary sample (EnrollmentService, course enrollment on Marten) in the full dev fleet and a new dcb scenario: tagged events, a decision across two tags, tag-fed projections and a [DeciderFunction] handler. (#1651)

Changed ​

  • The push splitter also divides on uncompressed size. CritterWatchOptions.MaxPushRawBytes (default 12 MiB, three quarters of the console's 16 MiB decode cap) bounds each push's payload before Brotli, beside the wire budget MaxPushWireBytes. It applies even with the wire budget off. Highly compressible shard-state pushes (60x and more on large tenanted fleets) could fit the wire budget and still be rejected whole by the console. A single piece that cannot be divided further logs an Error naming the decode cap. Needs Wolverine.CritterWatch 1.4 on the service. (#1671)

Fixed ​

  • Composite member shards of the live version read Paused when the warming-up version of the same composite was paused. A member names its composite without a version. During a blue/green warm-up the console matched it to the newest composite row, so a paused green V48 composite painted every blue V43 member Paused, in the shard list, the Paused count, needingAttention and list_shard_health. A member now follows the live composite at its own version when there is one. (#1679)
  • The Dashboard and the Services page disagreed on how many services were healthy. The Dashboard's Total Services card counted client-side alerts only. A service whose only alert was raised by the server, ServiceOffline among them, was healthy there and critical on Services. Both count with one function now. The service page's Health card read DLQ/hr from the metric counter, under a tile reading the table-derived rate. It reads the same canonical rate now. Top 5 DLQ Services says its figures are stored counts. (#1656)
  • Agent counts disagreed between the Services row, the Cluster tab and Node Detail. On a cold load of a service or node page, the agents read could answer before the first snapshot and was dropped when the snapshot created the service. The page then read 1 node · 0 agents and No agents on this node beside a Services row counting the right number. (#1655)
  • Node Detail on a node the console does not list now says the node is not running and that Eject removes any record Wolverine still holds for it, and links back to the service's nodes. Eject stays, because it acts on Wolverine's own node table, which can still hold a node the console no longer shows. (#1657)
  • The TeleHealth sample's BoardHistory compaction policy read Failing on every fleet run. It had no scope its store could read. It is removed; BoardHistoryOneTenant still shows a working dry run. (#1650)

Stack: Wolverine 6.64.0, JasperFx 2.102.0, Marten 9.59.0, Polecat 5.50.0, Fisher 1.29.0, Weasel 9.50.0 (unchanged from 1.3.0).

1.3.0 ​

Released 9 November 2026 — restart a circuit-broken listener from its alert, document storage sizes, an hour of projection trend, console reads gated like their MCP twins, and the projection screens at fleet scale.

Read Upgrade Notes before upgrading: with RBAC on, four console reads now need a grant, and Wolverine.CritterWatch 1.3 on a monitored service brings Wolverine 6.64 with it.

New ​

  • Restart a circuit-broken listener from its CircuitBreaker alert, and see how it came back. The alert now records the breaker's threshold, pause time and when the breaker will restart the listener by itself. The restart's confirmation says how long that is ("restarts it by itself in 3m 12s"), so you can choose to wait. The alert resolves when the listener is back at Accepting, and says whether an operator restarted it (who, and from the console or MCP), the breaker did after its pause, or the application started it itself; the signal is Wolverine 6.64's own. A trip within 5 minutes of an operator restart re-raises the alert marked re-tripped after operator restart: the cause is still there. Over MCP, restart_listener takes an optional alertId and reports whether the listener reached Accepting and whether it re-tripped, rather than only "accepted". The restart itself now uses Wolverine's RestartAsync(force: true), which rebuilds a stuck transport listener and, on a durable local queue, cancels the breaker's pending restart instead of racing it. Needs Wolverine.CritterWatch 1.3 on the service.
  • Document storage size and row estimates per document type, on demand, on the Document Store Explorer's overview, beside the exact counts 1.2 added. Read from the database's catalog (PostgreSQL, SQL Server; SQLite where dbstat is available, otherwise the file's size), labelled as estimates, with when they were read. Needs Wolverine.CritterWatch 1.3 on the service.
  • An hour of backlog per projection on the Operations tab's Projections grain: a small trend line under the rate and ETA, green while the backlog closes and red while it grows, so "behind but catching up" reads differently from "behind and falling further".
  • Projection lag measured in time. Beside the backlog in events, the Projections grain shows how old the oldest event a projection has not processed yet is (oldest pending 3m 05s), from its worst shard, with the tenant or database it is waiting on. The Shards grain has an Oldest pending column you can sort by, the projection page shows it per projection and per shard, and the MCP get_projection_lag and list_shard_health tools return it. Within a health level the Projections grain now sorts the longest wait first. Needs Wolverine.CritterWatch 1.3 on the service and Marten 9.59, Polecat 5.50 or Fisher 1.29; elsewhere the lag stays in events.

Security ​

  • Console reads of business data ask the same capability as their MCP twins. On a console with RBAC, the Conversations page needs the new conversations.view; dead-letter messages, failure groups and re-failures need dead-letters.view; a stream's events, tag queries and the By Filters query need event-store-explorer.view; and the MCP tool get_command_outcome needs mcp.conversations.read. A refused read says which capability it needs, never an empty list. Nothing changes without RBAC. See Console reads of business data.

Fixed ​

  • Adding AddCritterWatchHttp() to a service the console already knew never reached the console. The capabilities hash left out the ASP.NET surface, so the service's re-announcement was dropped and the HTTP Explorer kept saying discovery was never enabled until the service was evicted. The same was true of gRPC methods, sagas, document stores, DbContexts, tenant sources and tags. All are in the hash now.
  • The conversation hop table was sorted on every batch. Keeping the newest 50,000 hops cost a full sort of the table after every ConversationHopsCaptured batch, inside the lanes that also carry service telemetry, and was most of a busy console's database load. One node now trims it once a minute, deleting only the oldest overflow.
  • Every projection read Stalled while it advanced at hundreds of events a second, with one frozen lastAdvancedAt for all of them. The 1.2.2 client could hold a shard's advance back indefinitely when it batched telemetry; it now keeps it. The console also counts a report that moves a shard forward as an advance, the row's last advance is the newest over its shards rather than the quietest tenant's, and a Stalled reason names the shard it is about.
  • A composite's blue/green rollout read badly in three places. The rollout panel shows one block per composite with its members behind an expander; a grid row holds only the version it links to and says V41 · also V43 during a rollout; and the grid's rate and ETA are the rollout panel's, per version, so the two screens agree.
  • The projection detail page on a per-tenant sharded projection showed one anonymous shard with a gap that described nothing. It now has a header from the same server row as the Operations grid, a worst-shards table paged and sorted on the server with tenant and database filters, a version switcher during a rollout, and the composite's members with their own roll-up.
  • A paused projection showed a rate and an ETA for five minutes after the pause. The rate was measured over the window that ended at the pause, so the row read, for example, 3/s · ETA ~3m while its backlog grew. A paused, stopped or unassigned shard now reads 0/s, with no ETA.
  • The Document Store Explorer's Overview tab could not be opened while a service was selected: it switched straight back to Documents. It opens now, and a table with no row estimate reads no row estimate.
  • GET /api/critterwatch/timeline could fail with an empty 500 indefinitely after 1.2.2, while the store tried and failed to build its new indexes on first use under a 30-second timeout. They are built at startup now, with no timeout; on PostgreSQL in the background, answering 503 with Retry-After until they exist.

Improved ​

  • No screen loads every shard row of a service any more, except the Operations tab's Group by tenant view when you turn it on. The service page's Cluster tab, the subscription detail page, the Step Through tab, the standalone Projections page and the Raw data page each read only what they show.

Stack ​

Wolverine 6.64.0, JasperFx 2.102.0, Marten 9.59.0, Polecat 5.50.0, Fisher 1.29.0, Weasel 9.50.0.

1.2.2 ​

Released 4 November 2026 — the Operations tab at fleet scale: the Projections grain no longer loads every shard, the timeline answers at once, and composite member rows stay current.

Fixed ​

  • The Projections grain on the Operations tab loaded every shard row of the service before it showed anything. On a service with 516 databases and 2,173 tenants that was 150,628 rows in about 300 requests, and every row read Unknown until a walk that did not finish. Each row now comes from the server in one request, built from the same shards as the tiles above it, so the grid answers at once at any size. Expanding a row loads that projection's own shards for its tray. Group by tenant still loads every shard, because it breaks every projection down by tenant, so it now does so only when you turn it on. See The Operations tab.
  • Composite member rows went stale on the console, holding positions days old while the database had them at the high-water mark. The members then read as behind and their composite as Unknown. On a large fleet the service holds its telemetry and sends it in one push, and a 5,000-item cap on that push dropped the oldest items. Member positions are sent only when they change, so a dropped one was not sent again, and the same members were dropped from every push. The cap now trims only metrics samples, never a shard's state; a push too large for the broker is divided as before. Needs Wolverine.CritterWatch 1.2.2 on the monitored service. A service affected by this logged "exceeded the 5000-item cap after coalescing".
  • A projection that moved from Async to Inline left progression rows that read Stopped or Catching-up forever, and an 18-day-old position could win Max Lag and turn the service's lag verdict to "behind". Those rows now read as retired. They no longer feed Max Lag or any verdict, on any client version. With Wolverine.CritterWatch 1.2.2 on the service they also collapse into the Retired versions panel and can be ejected, which the service previously refused because the projection was still registered.
  • GET /api/critterwatch/timeline took 18 seconds on a large console, on every Dashboard and Operations load. The timeline had no index, so every read sorted the whole table. It now has one on its timestamp and one on service and timestamp, in all three store flavours.
  • projection-rollup was four times slower than the summary over the same rows on a fleet with composite projections. Attributing member rows to their composite grew with the square of the composite's size; it is now linear.
  • The Retired versions panel filled the first screen of the Operations tab. It now sits below the grid as one line, 39 collapsed versions · 58,848 progression rows, with Review to list them, and several can be ejected behind one confirmation. Each version is still ejected and checked on its own, and one summary reports the result.
  • Narrowing the paged shard query to a store matched nothing when the store was given without the trailing slash the service reports it with. It now matches either spelling.

Improved ​

  • Rate and ETA on the Projections grain. A projection that is behind shows how fast it is catching up and when it should be level, 720/s · ETA ~53s, or not moving. These are the figures the rollout panel already showed, now outside a rollout too. Within each health level, the largest gap sorts first.

1.2.1 ​

Released 3 November 2026 — the MCP projection stepper works again on a console that enforces RBAC.

Fixed ​

  • run_projection_stepper was refused on every console that enforces RBAC. 1.2.0 made the replay it sends to the service need projection-stepper.view, and the tool still sent it without the caller, so the console saw nobody and refused it whatever the agent was granted. The tool then waited out its 45 seconds and answered NoResponse. Consoles with RBAC off were not affected, and the console's Step Through tab always worked. The tool now sends the caller, so the grant is checked against the agent.
  • query_events described its inputs wrongly. Its where example used DateTimeOffset.Parse, which the predicate grammar refuses: only string and collection methods may be called. A quoted date compared with a date member is read as UTC instead, as in Timestamp > "2026-10-27". And leaving out tenantId reads the default tenant, not every tenant, so on a conjoined store name the tenant. The Event Explorer page had the same date mistake.
  • The RBAC page miscounted. It said six MCP read gates and 49 operator actions; there are seven and 52. It also listed a document filter watch family among the MCP action tools, which has no MCP tool.

1.2 ​

Released 3 November 2026 — the explorers: an Overview of every event and document store, Step Through, saga instances, the Schedule Explorer and the Event Model, plus AI dead-letter explanations, paging and listener commands that reach every node.

New ​

  • A conversation starts at the HTTP request. A conversation begun by a Wolverine endpoint now opens with a card for the request, labelled with its method and route, and no OpenTelemetry is needed. Only endpoints that caused a captured message get one, and CritterWatch samples nothing to do it. See An HTTP request is the entry.

  • Watch a saved document filter. Choose the bell beside a saved Document Explorer filter to turn it into an alert on a business condition: a threshold on its count (above, below or changed), a severity, and how often to count. The console runs the same store-side count the page runs, on the monitored service, and raises an ordinary alert (DocumentFilterWatch) that Slack, webhooks, paging and AI triage already handle. The dialog shows the store's plan and the index advice before it is created; a watch cannot run more than every 5 minutes, a store has at most 20, and creating one needs the new documents.watch capability. A service that does not answer, or a filter the store now refuses, is its own DocumentFilterWatchUnmeasurable alert and never resolves the condition. See Watching a filter.

  • AI explains a dead-letter failure group. On a console with LLM triage set up, a failure group's Explain these failures action asks the model what its dead letters have in common: a summary, the probable cause, whether they look Transient or Permanent, and a suggested next step. The answer is stored with the group, labelled suggestion only, not a diagnosis, and a suggested replay or discard marks the group's existing action rather than adding one. It runs only when asked, needs the new dlq.triage capability on each of the group's services, reuses an unchanged group's answer for 24 hours, and never sends message bodies unless DeadLetters.IncludeMessageBodies is set; other headers than Wolverine's own go by name only. See Explaining a group with AI.

  • Sagas for agents. Three MCP tools answer "where is this saga instance, and what is it waiting on?": list_sagas (the sagas each service runs and the messages they handle), list_saga_instances (recent instances from the service's own saga storage) and get_saga_instance, which returns one instance's state, the messages it has executed, the timeouts it has pending and the messages still outstanding, with every source saying whether it answered. They need the new mcp.sagas.read capability. list_conversations and query_scheduled_messages also take a sagaId. Pending timeouts need the service on Wolverine.CritterWatch 1.2, which filters its scheduled messages by saga id (Wolverine 6.59); an older service is not asked, and the answer says so. See MCP › Sagas.

  • Agents can write and read a failure group's note. The new MCP tool set_dead_letter_group_note writes the same note as the console's pencil (dlq.note, audited as MCP), so an agent can leave what its investigation found for the next operator. summarize_dead_letter_groups now shows each group's note, author and time; it had a note field that was always null.

  • AI alert triage reaches Slack and the webhook channels. When LLM alert triage stores an assessment, the Slack package replies with it in the thread of each alert it covers, and each notification channel whose severity allowlist admits the alerts gets one CritterWatch AI triage post. Every destination labels it "suggestion only, not a diagnosis" and names suggested actions without offering a button. SlackConfiguration.IncludeTriage = false keeps it out of Slack. A custom IAlertNotificationSink receives it through the new, optional NotifyTriagedAsync.

  • An EF Core entity's CREATE script. The EF Core Explorer shows the script that creates an entity's table, in an expanded entity and in the Entity configuration drawer over its rows. It is fetched when asked for, like a document's DDL, and it is the model's script, so a database behind on its migrations will differ. A view, a provider with no SQL and an entity with no table each say why they have none. See CREATE script.

  • A projection links to everything it writes. An event projection and a composite now link to their documents, as a single stream or multi stream projection does, on the Projections grid, the projection's page, the Configuration tab and a read model card. One projection can show several links, and a composite's row carries its members'. An entity written through one DbContext opens the EF Core Explorer, and the Configuration tab now offers that link too. A type with no matching document or entity gets no link, and a Live projection or a subscription gets none. See From a projection to its documents.

The explorers ​

  • Event Store Explorer › Overview. The explorer opens on one table of every service and event store in the fleet: its engine (Marten, Polecat or Fisher, as an icon), the store's name as registered, its tenancy, how many projections, subscriptions and event types it carries, and its alert count beside the service name. Click a row to drill into that store; Back returns to the table. See The Overview tab.

  • Step Through is a tab of its own. The Projection Stepper and Rehydrate Aggregate share one standing tab on the Event Store Explorer, which also says what each mode does and every way to reach it. Streams, the Event Explorer, the Operations grid, a projection's page, a projection's alerts and dead letters, and a view's card on the Event Model all hand off to it with the projection and the source already filled in. See Step Through.

  • Query events by their bodies and their metadata. The Event Explorer takes a WHERE and an ORDER BY over event bodies and over event metadata on all three stores. A WHERE on metadata the store does not capture is refused with the reason rather than answering nothing. By Filters is now By Filters or Tags, because it queries by DCB tags too, and a query that matched nothing says No events match. A body-only result is a grid of bodies, not events with an invented sequence, version and timestamp. See WHERE / ORDER BY over event bodies.

  • The Streams tab and the Event Explorer query only when asked. Opening a tab or changing the scope sends nothing, which matters on a store of millions of streams. The Streams tab filters by stream type and can count how many streams match. See The Streams tab.

  • Every tenant of a conjoined store. The Event Explorer reads every tenant of a conjoined store, and asks for a tenant where a database-per-tenant store needs one, instead of reading only the default tenant. The store picker scopes the whole Operations tab. A service that refuses the explorer shows the refusal rather than an empty store, and a shard's age reads from when it last advanced, so Stalled needs a shard behind a mark that can be measured.

  • Document Store Explorer › Overview. Every service, store and document type in one tree or table, with an exact row count per type or per store when you ask for one. Counting reads each table, so it is never automatic, it can be cancelled, and each count says when it was taken. See Overview tab.

  • Is a document type still healthy? On the Documents tab, Health runs the oldest and newest documents of a type through the service's own serializer and reports the ones that no longer deserialize, then compares the shape of the older and newer halves (members missing, added, or holding a different JSON type). Tenants counts a type per tenant, largest first. The type's drawer shows its mapping configuration and fetches its DDL on demand. See Documents tab.

  • One saga instance, in full. A saga instance's page shows its state from saga storage beside the messages it handled, its spans and its pending timeouts. A Saga flow draws what the saga declares beside what its instances were seen to do, and marks where the instance you are looking at is now and what it is waiting for. Instances can be searched where the saga's storage allows it, with a preset for instances that look stuck. See Saga Explorer. Agents get the same through the new saga MCP tools above.

  • The Schedule Explorer tells you what a schedule is doing.

    • Every cron pattern has a plain-language label beside it, in the schedule's own zone.
    • A schedule whose service is gone reads Service down and drops a dead owning node.
    • Trigger now says when it will run on the schedule's execution path.
    • A command with no answer releases its row after 30 seconds and says so.
    • A compaction policy's page can run it, and its recent runs fold a repeated outcome.

    See Schedule Explorer.

  • One display time zone. The header chooses the zone every time in the console is shown in, UTC by default, and every date picker reads and writes in that zone too. The console still stores UTC; this changes only what you see. See Display time zone.

  • Conversations: find one by anything, and go from a span to what it names.

    • Search by any id, service or message type.
    • Each span opens a pop-up with its tags, events and links: a handler or HTTP endpoint to its chain, a gRPC method, an Event Model slice, a failed span to its dead letter, and a message to the events it appended.
    • The span table sits under the causal graph, with two-way hover.
    • With a store's connection tracking on (Marten's TrackConnections), the handler, saga and stream tags and a cascaded message's parent are read through the connection span, so the conversation is no longer drawn as inferred and blamed on an HTTP request. See Store spans.

    See Going from a span to what it names.

  • The Event Model is its own screen. You choose a service first and see every slice count on arrival.

    • Message, command and event types can be included or excluded.
    • Each card drills down to what it shows.
    • A command reached by both HTTP and a message shows both entry points.
    • A slice with no pattern reads unclassified and can still be reached.
    • The spec UI appears only when a spec monitor is configured and reachable.

    See Event Modeling.

  • EF Core Explorer. An Overview of every DbContext in the fleet, deep links, an entity drawer with HasIndex advice, and an entity's rows browsed through the Document Explorer page. See EF Core Explorer.

  • The sidebar says where each link goes. Every link in the left navigation has a tooltip describing the screen it opens; in the collapsed rail the tooltip leads with the link's name.

Operations ​

  • Paging to PagerDuty and email. Two new notification channel types page an on-call engineer from an alert. See Paging: PagerDuty and email.
  • Runtime pressure. Each node reports its process CPU and managed heap, which Service Details shows per node, and a node at or above 90% of either for five minutes raises NodeRuntimePressure. See Runtime pressure.
  • A Grafana board pack. A plain Grafana dashboard over the same Prometheus series the console reads, for teams that already watch rates in Grafana. See Grafana boards.

Fixed ​

  • A dead letter time range means when it failed. The Dead Letters page, Failure Groups, the summaries and the MCP dead-letter tools all ranged on when a message was sent, so a message read again from a broker topic with an old sent time was missing from "the last 24 hours" a minute after it failed. They range on the failure time now, and so do Replay All, Discard All and the operations on a failure group, so an action touches what the page counted. The Messages tab shows Failed At with the sent time under it, and query_dead_letters returns failedAt and lists the newest failure first. This takes Wolverine.CritterWatch 1.2 on the service: an older client can only range on the sent time, and the page names those services in a warning while a range is set. The MCP tools say the same with rangedOn: SentAt.

  • A projection's side effects started a conversation of their own, and it read as complete. With JasperFx 2.99 a message published from RaiseSideEffects carries the correlation id of what appended its event, when the store captures correlation ids. So it joins that conversation, its HTTP request included, and two flows the daemon reads in one page stay two. The id crosses and a span does not, so the edge is dashed, and a notice says the event and the projection between them are not drawn. A conversation that still starts at such a message, on a store with no correlation metadata, now says so too, and get_conversation reports it as unlinkedSends. A timer or a background loop looks the same on the span, so its conversations carry the notice as well. See A projection's side effects.

  • A command a service refused read as success. The timeline entry for a listener action was written when the console sent the command, so a restart the service refused still read "Listener resumed". Those entries now read "Listener resume requested", and a failure the service reports lands beside them as a warning in a new Command category. Over MCP, accepted: true meant published and a refusal ended in the service's log. Every action tool that relays a command now returns a correlationId and hostOutcome: "unknown", and the new get_command_outcome tool reads back any failure a service reported, including a license refusal on a projection, schedule, compaction or high water command. It can never report success, because only failures are recorded. See What happened to a command.

  • Throughput means executions on every metrics source. Prometheus and VictoriaMetrics read MessagesPerHour from the sent counter and compared it with a baseline built from executions, so a service that fans out could read as a spike and a consume-only service read near zero. Application Insights counted received messages as executions, which left out local queues. Expect MetricsThroughput to fire differently on those sources after upgrading, and on Application Insights expect the failure rate and the 24h trend to move too.

  • Pausing a listener reaches every node. A pause by endpoint URI used to stop only the node that handled the command, so a follower went on consuming while the console said paused, and a restart left a node that joined during the pause latched. PauseListener and RestartListener now act on every running node, an exclusive listener only where it runs, and a node that does not apply the command is named in a CommandFailed. Pausing a buffered local queue, which has no way to stop, is refused rather than reported as done. See what the pause reaches.

  • A listener command for an endpoint the service does not have says so. A pause, restart or drain naming a URI no listener answers to, a mistyped queue for instance, was accepted and reported nothing, so it read exactly like a pause that worked. It now fails with a CommandFailed naming the URI. A restart still lifts any pause persisted for it first, so a pause left behind by an endpoint a deploy removed can be cleared.

  • Console actions check what their MCP twins check. Acknowledging, snoozing or clearing an alert, evicting a service and deleting metrics samples now need the same capability from the console as over MCP, and deleting metrics samples now needs a license as well. Pinning a service to a licensed slot needs the new service.license-slot. Step Through needs projection-stepper.view, and each run is recorded in the audit log. AI alert triage needs a license, as the dead-letter explanation already did. A refused action says why instead of doing nothing. See Capabilities.

  • Restart All lifts persisted pauses. RestartAllListeners started the listeners on one node and left their restrictions in place, so a listener paused on its own read as resumed and came back latched on the next boot. It now clears every persisted listener pause in one step and restarts listeners on every node. Paused projections stay paused.

  • A console with LLM alert triage configured could not start. 1.1 documented AddCritterWatchAlertTriage and shipped a standalone console that threw MissingPreBuiltTypesException naming LlmCallout at startup as soon as it was called: the console runs Wolverine's static codegen, which had no handler for the callout. Wolverine 6.59 runs callouts on a pre-built handler under static codegen, and 1.2 takes it. DeduplicateCallouts is not available on a standalone console for the same reason; see Cost and rate control.

  • Alert triage sent the model its context as an escaped string. The alerts, their recent history and the fleet's health were built as JSON and then serialized a second time, so the model received one quoted string full of \" instead of a JSON object. It could usually still read it, at a cost in tokens and reliability. The context now arrives as JSON, for alert triage and for explaining a dead-letter group.

  • Seeded dead letters read as real failure groups. The chaos seeder (seed_dead_letters, POST …/chaos/{service}/seed-dead-letters/{count}) built its exceptions without throwing them, so every Failure Groups title read "…: no stack trace". Each seeded failure is now thrown from a site of its own, and its group is titled after it, for example TimeoutException: DeadLetterSeeder.callPaymentGateway(...). The seeded dead letters were always fingerprinted into one group per exception type; only the titles change.

  • Slack follow-ups lost their thread after a restart or on another node. Where each alert was posted to Slack was kept in memory, so after a console restart, or when another node of a clustered console handled the follow-up, the "elevated" and "resolved" posts became new top-level messages. It is now stored in the console's database (a new AlertNotificationThread document; see Upgrade Notes if your database is scripted). Alerts posted before the upgrade still post their follow-ups on their own once.

  • Per-message-type alert thresholds now fire. Thresholds saved for one message type on the Alert Configuration page, and For<T>() declared baselines, were stored and applied nowhere: every metrics alert read the service-wide values. Each message type with thresholds of its own now gets its own execution-time and failure-rate alerts, against that type's numbers and baseline. This works on the console's own metrics and on Application Insights. Prometheus, VictoriaMetrics and Datadog cannot narrow a query to one message type, so there the thresholds are still not evaluated, and the console logs that once. See Alerts › Cascade order.

  • advance_high_water_mark checks the skip count it asks for. Its description said confirmSkipCount must equal the preview's missingSequenceCount, and nothing compared them. The service now recomputes the gap just before advancing and refuses, writing nothing, when it differs from the count the operator confirmed. The console's Advance mark… dialog sends the count you typed. A service still on Wolverine.CritterWatch 1.1 ignores the count and advances on the mark check alone; the tool's result says so (skipCountEnforced: false).

  • The Event Model reads [DeciderFunction] handlers. The CritterWatch source generator now recognizes Wolverine's store-agnostic [DeciderFunction] on a class or a method, as it does each store's [AggregateHandler], so those handlers and the aggregates they write appear on the Event Model. A method-level [AggregateHandler] now names its aggregate too.

  • restart_listener left a listener stopped on a Balanced host. Since Wolverine 6.48, restarting a listener addressed by its endpoint URI waited for an agent confirmation that never came, because an endpoint is not an agent. The pause row was deleted and the listener stayed stopped, with nothing to say why; a second restart worked. A restart now resumes it on the first try, on every node.

  • An expired session was a white page. When a login proxy redirected the console's own script requests, the first page's code failed to load and nothing rendered. The console now shows a readable notice that the session has expired, with a reload.

The Critter Stack underneath ​

  • Wolverine 6.61.0, JasperFx 2.99.0, Weasel 9.48.0, Marten 9.57.0, Polecat 5.48.0 and Fisher 1.27.0.
  • Fisher 1.27.0 leaves archived events out of every default event read.

1.1.1 ​

Released 28 October 2026 — two fixes for the SQL Server and SQLite consoles. PostgreSQL consoles are unaffected.

Fixed ​

  • Agent assignments froze on SQL Server and SQLite consoles. Every agent start, stop or move after the first one a service reported was refused with JasperFx.ConcurrencyException: Optimistic concurrency check failed for CritterWatch.Services.Model.AgentRoster, retried, and discarded. Service Details and Node Detail kept showing each service's agents as they were first reported, and the console logged the failure for as long as services ran. Polecat and Fisher treat a revisioned document's own version as the revision it must be newer than when the document is stored again, and Marten does not, so the same code worked only on PostgreSQL. The console now lets the store assign the next revision. No action is needed: each service's roster corrects itself on its next report.
  • A SQL Server console first created by a 1.0.0 alpha could not record metrics. Its metrics table predates partitioning, and every version since 1.0.0-beta.1 failed each metrics flush with "column 'bucket_end' cannot be added to an existing table". The console now migrates that table at startup and keeps the metrics inside your configured retention. See Upgrade Notes.

1.1.0 ​

Released 27 October 2026 — the first feature release after 1.0. It makes the console usable on fleets of hundreds of databases, turns the Event Model into a live screen rather than a diagram, lets a rebuild be scheduled for the middle of the night, puts recurring cron schedules on a screen you can pause from, lets the whole console run inside your own application, and opens the event store to AI agents. It also carries every fix from the unreleased 1.0.2.

New since 1.1.0-rc.3

Before upgrading from rc.3, read Upgrade Notes. Failure groups add two capabilities and two console documents, Conversations for agents adds a third capability, and failure groups need Wolverine.CritterWatch 1.1.0 on the service: an rc.3 service is treated as 1.0.

  • Failure groups. A new Failure Groups tab groups dead letters by failure across the fleet, and a whole group can be replayed, archived, restored, discarded or cancelled. Groups carry notes, and a finished replay says which messages failed again. See Dead letters as failure groups.
  • Conversations for agents. list_conversations, get_conversation and cw-conversations give an agent the same causal graph and timeline the Conversations screen draws. See AI and MCP.
  • A new package for .NET Aspire, CritterWatch.Aspire, replacing Aspire.Hosting.CritterWatch.
  • The broker-less HTTP channel works. As documented through rc.3 it could not: see Fixed and Upgrade Notes. HTTP and gRPC are now both tested end to end.
  • Wolverine 6.56 and JasperFx 2.96, with the same Marten, Polecat and Fisher versions as rc.3.
  • A manual review of every new feature area against a 32-service fleet, and the fixes it produced: alerts and projections on stores Wolverine does not manage, the Event Model's handler names and its disagreement markers, document edits on projected documents, recovered messages in Conversations, and more. See Fixed.

New in 1.1.0-rc.3

Before upgrading from rc.2, read Upgrade Notes. In particular: if your console's database is scripted rather than created by the console, run db-patch first, because rc.3 adds five tables.

rc.3 is built on Wolverine 6.54, with the same JasperFx, Marten, Polecat and Fisher versions as rc.2. What it adds, most of it described further down this page:

  • Dead-letter management, followed to the end. A new Operations tab tracks every replay, discard and edit across every database it touched, with counts from Wolverine 6.54 and each failed database's error. Every one of those actions is now audited. Bulk actions act on exactly the set their confirmation counted, and can carry a reason. The message drawer shows a dead letter's replay history. Edit & Replay needs dlq.edit. See Follow a replay or discard to the end.
  • AI agents ask before they act. The MCP server now requires a confirmation token before an irreversible tool runs. Every tool states its read-only and destructive hints, a read-only switch registers only the read tools, and audit rows say whether an action came from the console or an agent, and which client. See AI and MCP.
  • Documents. You can edit, delete and restore documents from the console or over MCP. Explain shows how the store ran a query, and a read-only index advisor shows the C# that would index a filter which scans. Filters, history and column layouts are saved, and you can compare a projected document against a live rebuild of its stream.
  • Schedules. Each schedule has its own run history and detail page, and its own dead letters. Schedules driven by timers show up too.
  • Alerts. New alerts for an agent the leader could not place (AgentUnplaced), an overloaded node (NodeOverloaded), and a high water mark held behind committed events (HighWaterMarkFrozen). Per-tenant alerting now works on Application Insights too. See the Alerts reference.
  • Projections. You can advance a high water mark past a gap, with the evidence shown first. The members of a composite projection appear as rows of their own.
  • Services still on 1.0. A service on the published 1.0.1 client registers and reports to an rc.3 console. That is now checked before every release.
  • Fixes. The Save & Replay button and fourteen other operator confirmations that never opened, Alert Config thresholds that did not save, an agent's dead-letter actions refused on a host with RBAC, and more. See Fixed.

The documentation was reviewed page by page against the code for this release.

Upgrading from 1.1.0-rc.1

If you embed the console with role-based access control turned on, upgrade. In rc.1, a command sent from the browser to an embedded console reached your application's handlers without passing through the console first. The console's own checks were skipped: RBAC, the audit log, the "command accepted" acknowledgement, and the refusal for a service parked over its licensed limit. A replay denied by RBAC went ahead, and a pause wrote no audit row. In rc.2 every command goes through the console and then runs your application's handler exactly once. Standalone consoles were never affected.

More embedded fixes in the same family, where the console was counted as part of your application:

  • Event Store Explorer and SQL view. The SQL view refused every statement as ambiguous between "two event stores", because the console's own store was counted beside yours. It now runs against your store. The Document Explorer had the same problem and is fixed too.
  • Dead letters and scheduled messages. Operations no longer fan out to the console's own SQLite file as though it were one of your databases.
  • Metrics, agents and timeline. Your metrics no longer include the console's own message traffic, and your agent list and timeline no longer show the console's background agents.
  • Replies. Each reply from the console arrives once, not twice.
  • SQLite. Running a SQL statement no longer leaves a pooled connection read-only, so your application's next write no longer fails with "attempt to write a readonly database".
  • Memory in tests. A disposed host is now released. The console's update check held every host that a test suite built and disposed, about 18 MB each. A second, independent hold comes from the projection daemon itself; it is fixed in JasperFx 2.90.2 and later, and rc.2 depends on 2.91.0.

A new package for .NET Aspire: CritterWatch.Aspire. It is published from the release after 1.1.0-rc.3. It was built as Aspire.Hosting.CritterWatch, but Aspire.Hosting.* is a reserved package-id prefix on nuget.org, so it was renamed before it was ever published; the extension methods stay in the Aspire.Hosting namespace. From your AppHost: AddCritterWatch<TProject>() adds your console project with an "Open console" link and a health check on /health/live. WithCritterWatch(console) makes a monitored service wait for the console and gives it the console's license. WithJasperFxLicense() hands the license on as a secret parameter. The license matters because pause, rebuild and replay execute on the monitored service, and Aspire's child processes do not inherit the AppHost's environment. See Hosting.

Open a document type in SQL. The Document Store Explorer has a SQL tab now, and the Documents tab an Open in SQL link that opens it with select data from <table> where prefilled. The table name comes from the store's own DDL. It is the engine's own SQL (data ->> 'Name' on PostgreSQL, JSON_VALUE on SQL Server, json_extract on SQLite), run read-only on the monitored service with the same guard as the Event Store Explorer's SQL view. A service with document stores and no event store can use it too.

Schedule Explorer: how the last run went. Each schedule's last run now shows whether it succeeded or failed, the exception type when it failed, and how long it ran. A schedule whose last run failed reads Failing. An un-paused schedule reads Active instead of Running, which looked like "a run is in progress". Monitored services need the 1.1 client to report outcomes; an older one shows none.

Document Explorer: see what the store will keep before you save. The review step asks the monitored service for a dry run of the save and warns of any member the store would drop, or why it would refuse. MCP update_document takes dryRun: true for the same answer.

Dead Letters: filter by an envelope header. A Header filter matches a header exactly, so a recurring schedule's page opens on that schedule's own dead letters (recurring-schedule). Needs Wolverine 6.52 on the service.

Document Explorer: restore a soft-deleted document. A soft-deleted document's drawer offers Restore, and the delete confirmation now says whether a delete is soft or permanent, asking you to type the id only for a permanent one. MCP gains undelete_document.

A paused agent says so on the Agents tab. No agent ever showed its Paused or Pinned pill, and those filter chips always counted zero: the console compared the restriction against numbers while the service sends its name. A paused agent now reads Assigned · Paused rather than "Active — running on Node N", and an agent whose pause or pin is lifted returns to normal — before, it read Paused until it was rediscovered. Found by pausing a timer schedule, which now reads the same on the Schedule Explorer and the Agents tab.

New alert: NodeOverloaded. On a service with capacity-aware agent assignment, a node advertising load at or above its overload threshold for two minutes raises a Warning naming the node, and it resolves when the node recovers or leaves. It is the early sign of what AgentUnplaced reports once no node has room.

Watched tenants keep their metric rows. The per-service WatchedTenants list, which already kept a tenant's own projection-alert stream, now also keeps that tenant's per-tenant metric rows above PerTenantSampleCeiling (still 25 by default). The list spends from the same ceiling, so the storage bound is unchanged.

Per-tenant metrics alerts work on Application Insights sources, as they already did on Prometheus and VictoriaMetrics: each tenant is evaluated against its own measurements rather than the whole service's. On all three, a pass now costs a handful of queries per service however many tenants it has: each read is asked once, grouped by tenant, instead of once per tenant (about 35,000 PromQL queries every 30 seconds at 2,173 tenants).

Document Explorer: copy as C#, and the scope carries. An open document's Copy as C# copies class definitions inferred from it. The store and tenant picked in the Document Store Explorer carry to the Event Store Explorer for the same service, and back.

Document Explorer: newest first. A switch orders a type's documents by when they were last modified, most recent first. query_documents on MCP takes the same ordering as orderByMetadata: "LastModified desc".

Document Explorer: saved filters and history, kept by the console. Applied filters are remembered per type, and a filter can be named and starred. They live in the console's own store, so they survive a browser change; per user with RBAC on, shared per console without it. The grid's column layout — which metadata columns are on and the member-path columns — is kept the same way, per document type.

Document Explorer: what is in these documents? A Fields panel samples the type under the current filters and lists each member with how often it is present, its types and its commonest values. Clicking a value adds it to the filter; shift-click adds it as an alternative, and a number or date range offers its bounds. Right-click a value in the grid or the document drawer to filter on it or exclude it.

Document Explorer: will this filter use an index? Under the predicate box, a line estimates whether one of the type's reported indexes covers what the filter names, before it runs.

query_documents takes the metadata filters. The MCP tool now accepts correlationId, causationId and lastModifiedBy, as the Document Explorer does. A store ignores one of them on a document type that does not capture that column and answers unfiltered, so the tool's answer and the explorer both say so beside an applied filter (jasperfx#1077).

A page for each recurring schedule. Click a schedule's name on the Schedule Explorer for its definition, how its last run went, its last 20 runs across every node with how many failed, the Pause / Resume / Trigger now actions, and a link to its dead letters. list_recurring_schedules on MCP returns the same last-run outcome and run history. A pause or resume no longer blanks the "Last handled" column until the next report.

A service whose event store failed to describe itself now heals. When reading one event store's description failed (seen once under heavy load on an embedded SQLite application), Wolverine reported the service as having no event stores at all. The console kept that until the service restarted, because nothing made it ask again. The service now checks its own capabilities and resends them, backing off from 15 seconds to 4 minutes, until every event store it holds is listed. It logs a warning for each attempt, and an error if it gives up.

SQL view: every table in a FROM list is checked. The SQL view only runs statements over the store's own tables, and it checked only the first table after FROM or JOIN. So FROM mt_events, some_other_table reached any table the service's connection could read. Every comma-separated item is checked now, including items after a join or a derived table. Functions, derived tables and LATERAL items stay allowed, as before.

A versioned wire between services and the console. Every message a monitored service and its console exchange now carries an explicit name, fixed at the name it already had, so nothing changes on the wire today. Each push also says which wire protocol the service speaks. Both exist so a later console, built on the next major Wolverine, can keep reading 1.x services. A console that receives a newer protocol than it understands logs a warning once per service.

Hosts that turn off conventional local routing. With DisableConventionalLocalRouting(), messages CritterWatch sends to itself had no route. Embedded, the application got no metrics at all, and on any service, standalone included, stream compaction policies never ran. Every message CritterWatch sends to itself now has a route of its own.

Fisher / SQLite services. Wolverine.Fisher has no event-subscription agent family, so pause, restart and rewind cannot reach a projection on a Fisher service (wolverine#5094). The console now disables those actions with the reason, instead of offering them and then refusing. Rebuild still works.

Per-tenant subscription rewind. On a store that partitions its events by tenant, a rewind of one tenant now works. A rewind with no tenant fans out to every tenant's agent, wherever it runs. A rewind to a sequence number needs a tenant, because a sequence is a position within one tenant's events: the dialog asks for one, and a scheduled rewind is refused when you schedule it, not when it would fire. Needs JasperFx 2.91 and Marten 9.51 on the monitored service.

Restart high water detection. A new operator action restarts a node's high water detection loop when it has stopped polling. It never moves the mark. In 1.1.0-rc.2 a multi-database store whose projections Wolverine distributes was still left unread, although the Wolverine fix it waited for (wolverine#5092, Wolverine 6.51.2) had already shipped; those stores are now read too.

The Critter Stack underneath moves together to JasperFx 2.91.0, Wolverine 6.51.2, Marten 9.51.0, Polecat 5.43.0 and Fisher 1.21.2. Two Wolverine fixes worth knowing about:

  • wolverine#5087: a database-backed queue listener that stopped while holding messages could lose them.
  • wolverine#5092: under Wolverine-managed distribution, IProjectionCoordinator.AllDaemonsAsync() now returns only the daemons running on that node.

Monitored services should move their store with the 1.1 client

Relaxed in 1.1.0-rc.1. In 1.1.0-beta.7 and 1.1.0-beta.8 this was a hard requirement: the client depended on JasperFx 2.78, which added members every document store implements, so a monitored service on an older store restored and built cleanly and then failed at startup with a TypeLoadException naming LoadDocumentAsync. From rc.1 the client depends on a JasperFx that gives those members default implementations, so an older store starts again. Marten 9.43, Polecat 5.34 or Fisher 1.15 or later is still what the Document Explorer needs to read a document for a named tenant. See Upgrade Notes.

Upgrading from 1.1.0-beta.8

rc.1 makes the embedded console a guest in your application. Through beta.8, several of the console's settings reached the host that embeds it. The most serious one changed how your handlers fail. The console's retry ladders were registered host-wide, so every handler in an embedding application retried a failing message for more than six seconds and then discarded it instead of dead-lettering it. Those ladders now apply only to the console's own handlers. If you embedded an earlier beta, upgrade. The other leaks of the same kind are also fixed:

  • Message grouping. The console's ByPropertyNamed("ServiceName", "Id") rule took precedence over your own grouping rules for any message with an Id property. It now matches only CritterWatch's message types.
  • Durability. A host-wide policy such as UseDurableLocalQueues() no longer makes the console's relay-command queues durable.
  • JSON. The console's HTTP API always answers in its own format (camelCase, with enums as camelCase strings), whatever your JsonOptions say, and it no longer changes the enum casing of your own endpoints. With mapWolverineEndpoints: false, or a PascalCase JsonOptions, it used to answer in PascalCase. The About page then read undefined, and a settings save from the browser could store default values while returning a 200.
  • Tracked sessions. InvokeMessageAndWaitAsync, TrackActivity() and Bobcat acts ignore the console's traffic, so they settle with the console embedded.
  • Routing. The UI is served from endpoints under its mount path. Your SPA fallback (MapFallbackToFile, a catch-all route) no longer answers the console's deep links, /critterwatch/index.html is rewritten for the mount, and a missing asset returns a 404 instead of the index page.
  • What it reports. The application's capability card and Event Model no longer count the console's own projections, event types, endpoints and slices. A second MapWolverineEndpoints() no longer fails every request: the mount leaves an existing mapping alone.

There are two new options. UseCritterWatchEmbedded(configureEndpoints: b => b.RequireAuthorization("ops")) puts the UI, the HTTP API and the SignalR hub behind your authorization, and leaves your own endpoints alone. ConfigureMonitoring reaches the explorer switches (EnableDocumentExplorer, EnableSqlQuery, …) that a standalone service would set. See Putting the console behind your authorization and How your handlers stay yours.

A monitored service could crash while splitting a large push. The beta.8 push splitter recursed without bound on a push made of many small health reports and took the service down with a stack overflow. Every division now makes progress. If you set MaxPushWireBytes = 0 to work around the crash, you can remove that setting.

Retired projection versions stop piling up. On one console, 62% of all shard rows belonged to 38 retired projection versions, and nothing aged them.

  • An orphaned row is no longer rewritten on every full pass, and a version whose every row has been orphaned for 24 hours collapses into one Retired record that carries its shard count.
  • Eject a retired version, from the new Retired versions panel on the Event Store Explorer or with the eject_retired_projection_version MCP tool. Either one deletes every progression row of that version, on every database and tenant, on the monitored service itself. The service refuses if it still registers that version. This needs the rc.1 client on the service. An older client never answers, and after a minute the panel says so.
  • Phantom never-started shard rows left by earlier betas expire (keep FullSnapshotInterval under about 50 minutes), and the agent roster forgets nodes that have left the cluster.
  • Restart reports success only once the agent is running on a node. Before, it could show a green tick for a projection that was running nowhere.
  • A rebuild that runs longer than the acknowledgement timeout is no longer marked failed while it is still running. A second rebuild of a projection that is already rebuilding is refused, and the refusal is shown as information rather than as an error.

Alerts that never resolved now resolve.

  • Agent down is judged against the heartbeat cadence observed for that agent. On a fleet that heartbeats every four minutes, every agent used to read as down between heartbeats.
  • Per-tenant high-water mark stale no longer fires for an idle tenant while its database's agent is heartbeating. Alerts already raised this way resolve on the next evaluation.
  • A console self-ingest alert that was raised before a restart can now resolve after it.
  • An unhealthy agent says why: on the Agents tab, and in the broker status tooltip. The reason stays current, and an agent that leaves the cluster takes its health row and its alert with it.
  • One failure seconds after startup no longer reads as 156 errors an hour and marks the service Critical: rates are never extrapolated from a window shorter than an hour, and a dead-lettered message is no longer counted as two exceptions.

Telemetry and ingest. Three changes have upgrade consequences, all described in Upgrade Notes:

  • CritterWatch's own endpoints never use a durable inbox, whatever the host configures.
  • An application's own Metrics.Mode is kept. Code compiled against the old signature must be recompiled.
  • The store-version floor is a recommendation again, so a service on an older Marten, Polecat or Fisher starts. Document reads for a named tenant still need the newer store.

Also in this area:

  • Agent and endpoint health are sent when they change, plus a restatement every five minutes, rather than every 60 seconds.
  • Both ends log what telemetry costs, every five minutes: see Telemetry accounting and Ingest accounting. /ingest-health lists every ingest listener with its depth.
  • The discovery handlers that record message causation and event appends never duplicate an edge. On Fisher and Polecat they no longer collide with ingest at all; on Marten they still can, and retry. A retried attempt now logs at Warning rather than Error (per-handler failure log levels arrived in Wolverine 6.51, wolverine#5043), and one Error is logged only when an edge is finally discarded after its retries.
  • A late push from a service that is being evicted can no longer recreate its stream without a registration.

The browser holds less. Leaving the Operations tab releases a service's shard state after 30 seconds; on one fleet that state was 28 MB. The Message Log keeps full payloads only for the newest 25 entries, within 16 MB, and keeps a 2 KB preview of the rest. During an agent-churn storm, the dashboard's Recent Events stays populated. The storm itself is now written to the timeline as one row per node, AgentStopped entries can be compacted, and the timeline row cap per service defaults to 250,000.

The Event Model reaches the rest of the console. Handler and HTTP endpoint pages show the slice each one belongs to: its trigger, aggregates, declared events and the views they feed, its bound Bobcat specifications with their latest outcome, and its hotspots, declared-versus-code disagreements first. An event's page lists every slice that emits or consumes it. An HTTP trigger reads POST /api/orders and opens its HTTP chain, and says why when it cannot. Query slices render and filter, and aggregates show their identity type, as in Order (OrderId). See Event Modeling.

The Document Store Explorer filters and sorts on a document's own properties. Enter a WHERE and an optional ORDER BY in C# property names. The monitored service's store applies them through its own LINQ provider, so the console never builds SQL. A predicate that a store cannot answer correctly is refused, with a caret where it failed, rather than shown unfiltered. This needs the rc.1 client on the service. The MCP query_documents tool takes the same text, and the type picker groups sub-classes under their root. See Documents tab.

Also since beta.8:

  • Endpoints are labelled by their name, with the URI as secondary text.
  • db-dump, db-patch and db-assert now see every table the console uses. The new Database Schema and Permissions page covers what the console creates, when it runs DDL, and the smallest grant it needs.
  • Built on WolverineFx 6.51.0, Marten 9.50.3, Polecat 5.42.1, Fisher 1.21.1, JasperFx 2.90.1 and Weasel 9.45.1. The monitored-service package, Wolverine.CritterWatch, now depends on WolverineFx 6.51.0 and JasperFx.Events 2.90.1.

Upgrading from beta.8 is a package bump on the console and on the monitored services, with no manual schema steps on a console that migrates its own schema (the default; see Database Schema and Permissions if yours does not). Retired-version eject and document predicates need the services on rc.1.

Upgrading from 1.1.0-beta.7

Telemetry stays under the broker's message-size limit on the largest fleets. A monitored service sends the console its telemetry as one message per push, and on a fleet of a few thousand tenants and hundreds of databases that message was approaching Amazon SQS's 256 KiB cap. Past it, the broker rejects the message and that service's projections and topology stop reaching the console with nothing on screen to say why. MaxItemsPerPush did not prevent this, because it counts items rather than bytes.

  • Pushes are compressed harder: about 55–60% smaller on a large fleet's telemetry, for a few extra milliseconds per push on the monitored service.
  • A push that would still be too large is sent as several. The new MaxPushWireBytes option (default 192 KiB) sets the limit. The console merges the pieces, so nothing is lost, and the service logs once, at Information, when it starts dividing. See Push size.

Both changes are in the monitored-service package and need no upgrade ordering: consoles and services on either side of them read each other's messages.

Also since beta.7:

  • The console's 30-second refresh allocates less than half as much on a large fleet. Counting a service's agents by health no longer copies a string per agent: on a 75,000-agent service a quiet refresh now allocates about 9 MB instead of 20.

(Upgrading from beta.7 is a package bump on the console and on the monitored services, in either order.)

Upgrading from 1.1.0-beta.6

beta.7 is the large-fleet memory release. On a fleet of about 75,000 projection agents, one open browser tab could run the console out of memory: every connect built a 73 MB snapshot, half of it the agent roster, and wrote it as a single frame. The connect snapshot now carries five roster counts per service instead of the roster, and the Service Details and Node Detail pages load the agents they show when they open. Roster and agent-health updates go only to tabs that have one of those pages open.

The console also does far less work per telemetry cycle at that scale:

  • A node's agent-health report costs about 150 reads for 75,000 agents instead of two round trips per agent. A healthy agent's row is no longer rewritten on every report, and its alert stream is read only when its health changed.
  • The 30-second refresh reloads an agent roster only when it changed, and reads only the health rows and shard rows that moved. A quiet refresh tick at field scale allocates 46 MB rather than 273.
  • Matching a projection shard to the node running it no longer scans the whole roster per shard.

Monitored services must also move their store with this client: see the warning at the top of this section.

Also since beta.6:

  • The licensed service cap is enforced across the whole fleet. A console monitoring more services than its license allows now keeps the earliest registered (or the ones you choose with Keep monitored) and parks the rest: a parked service's telemetry is not applied and commands to it are refused, but its data is kept and it stays listed, tagged Not monitored — license limit. Before, only a brand-new service was checked, and a fleet already over the cap was monitored in full. See Licensing.
  • Blue/green rollouts no longer read as lag. While two versions of a service ran side by side, each reported the other's projection shards as retired, the shard rows flipped between the two views, and the warming version's whole gap was reported as the service's worst lag. A retired report no longer overwrites a shard a running agent is reporting, and retired shards are left out of the health counts and the lag. The Operations shard panel says how many shards are no longer registered.
  • The Document Explorer reads on the stores' shared contract: soft-deleted documents are hidden unless you ask for them (and then marked), All tenants reads every tenant explicitly, and reads are bounded (200 per page, a 256 KiB page budget, a 30-second timeout). Reading documents needs the new documents.read capability when permissions are wired, and every read is audited. See Upgrade Notes.
  • Per-tenant high-water marks carry their own liveness, so a partitioned store's tenants no longer inherit the staleness of a row the store has stopped writing.
  • An idle service registers. A service with no telemetry to send, such as one with a document-only store, never completed its handshake with the console.
  • The Event Explorer's Compact, stream fetch and query act in the selected tenant. On a store whose default tenant is disabled, a compact failed and an existing stream read as not found.
  • Stream compaction compacts per tenant on a store whose default tenant is disabled.
  • Recurring schedules have Trigger now, which runs a schedule once without changing its cadence.
  • Listener health shows the in-flight limit and partition-lane depth, and the idempotency guard's dropped-duplicate count, when they matter.
  • The Scheduled Messages page shows each message's tenant on a multi-tenant service.
  • Per-tenant metrics alerts work on Prometheus and VictoriaMetrics sources.
  • New MCP tool list_tenants, and a refused hard delete of a disabled tenant now says why.
  • A mapping DDL fetch that fails says so, instead of reporting that no DDL was generated.
  • CRITTERWATCH_DISABLE_SQS lets a RabbitMQ-only development BFF start without LocalStack.
  • Built on WolverineFx 6.43.0, Marten 9.43.0, Polecat 5.34.0, Fisher 1.15.0, JasperFx 2.78.0 and Weasel 9.38.0. The monitored-service package, Wolverine.CritterWatch, now depends on WolverineFx 6.43.0 and JasperFx.Events 2.78.0.

(No manual schema steps: the one new document, the license-slot pin, is created on first use. Upgrading from beta.6 is a package bump on the console, plus the store versions above on every monitored service.)

Upgrading from 1.1.0-beta.5

beta.6 fixes the Document Store Explorer on a service with more than one document store. It listed the document types of whichever store the service registered last, usually an ancillary one, for every store in the picker. A service's main-store types, whether registered up front or first used at runtime, never appeared, and browsing one of them returned an empty page instead of an error. Each store now answers for itself, and a request that names a store the service does not have, or names none on a service with several, gets an error listing the stores rather than a guess. The MCP document tools take an optional storeUri for the same reason. On Fisher, two document stores that share one SQLite file report the same identity and cannot be told apart (fisher#353): give each its own file.

Also since beta.5:

  • Evicting a service no longer stops it from ever coming back. Eviction archived the service's streams, and every store refuses an append to an archived stream, so the service's next push failed on every restart. Its history is now deleted, so it registers again the next time it reports, and alerts it raised before can fire again. A service evicted by an earlier version needs one more eviction; see Upgrade Notes.
  • A service whose every node has gone silent reads as Offline, with a new Critical Service offline alert. It only raises while the console is demonstrably receiving telemetry from other services, so it cannot fire on a console that monitors a single service. The Services page's health pill now reflects alerts raised by the console, not only ones reported by the services.
  • Two nodes running the same projection shard now raise a Critical Shard ownership conflict, naming both nodes. It triggers no recovery, because restarting a shard two nodes are running cannot settle which one owns it.
  • On a tenant-partitioned store, running per-tenant shards are no longer reported as orphaned. They were compared against a list that names projections without their tenant, so a healthy shard could read as retired and have its alerts suppressed.
  • Per-tenant projection dead-letter counts reach the console. They read zero on every store.
  • A failing database no longer empties the projection dead-letter page. The page shows every dead letter it could read and says how many databases could not be read.
  • Stream compaction on a store whose default tenant is disabled now selects streams for a policy scoped to a tenant. A policy with no tenant scope on such a store says to declare AllTenants or a TenantId. Compacting per tenant, as opposed to selecting, waits on the next JasperFx release; an armed run on a named tenant reports that it cannot compact yet rather than failing per stream.
  • An empty new tenant database is no longer "needing attention" until it has events to process.
  • The embedded console ships pre-generated handlers and never generates code at runtime. A Fisher console now creates its own event tables on a fresh database file; it could previously deadlock on its first write.
  • EF Core conjoined tenancy is read from the DbContext, and a dead letter from a conjoined tenant, replayed through the console, is handled for that tenant.
  • Built on WolverineFx 6.41.0, Marten 9.40.0, Polecat 5.32.0, Fisher 1.13.0, JasperFx 2.75.2 and Weasel 9.35.2. The monitored-service package, Wolverine.CritterWatch, now depends on WolverineFx 6.41.0 and JasperFx.Events 2.75.2.

(No schema changes. Upgrading from beta.5 is a package bump, plus one step for any service an earlier version evicted.)

Upgrading from 1.1.0-beta.4

beta.5 fixes the embedded console, which did not work in a real application. Mounted with UseCritterWatchEmbedded(), it served its page at /critterwatch and then went nowhere. Every screen, the dashboard included, is loaded on demand, and the browser asked for those screens' files at your application's root (/assets/…) instead of under /critterwatch, so it got a 404 and navigation stopped. The header logo and the transport icons were broken the same way. Everything the console loads now resolves under wherever it is mounted, and it keeps routing when your application's Content-Security-Policy blocks inline scripts. This is now tested in a real browser against the packed package, on every release.

Also since beta.4:

  • A browser tab that keeps losing its connection backs off properly. Reconnects that died right after reconnecting used to retry every second indefinitely, so one open tab on a very large fleet could keep the console rebuilding its connect snapshot. They now escalate to 30 seconds.
  • The ingest-health chip's count follows the live number while a back-pressure alarm is raised, instead of repeating the count from the moment it was raised.
  • On a tenant-partitioned store, shards are no longer reported as never started when their tenants have run them, and Inline and Live projections are no longer expected to have progress rows.
  • Roll-up alerts stop writing a membership event every evaluation cycle when only the "N of M" denominator moved and the set of affected tenants did not.
  • The Schedule Explorer shows a compaction policy as Failing as soon as the page loads, from the runs already recorded, rather than green until its next occurrence. Run now reports its outcome: the failure and its reason, or what a dry run would have compacted.
  • The Audit Log shows a SQL statement as one line with its outcome, and the SQL view states its 1,000-row ceiling as the contract.
  • A single-database store no longer leaves behind a frozen, database-less high-water row beside its live one. (A console upgraded from an earlier beta keeps the old row until its data is reset. It is harmless; see Upgrade Notes.)

(No schema changes, so upgrading from beta.4 is a package bump.)

Upgrading from 1.1.0-beta.3

beta.4 is the embedded-mode beta. The console now boots inside a Marten host on PostgreSQL queues with no broker, and that exact combination is under test for the first time — which found two things the docs now say plainly: the PostgreSQL transport needs role: MessageStoreRole.Ancillary or the host will not start, and the Marten registration takes an NpgsqlDataSource rather than a connection string (an earlier snippet had that wrong). Also since beta.3, four behaviours that had shipped without notes: the Conflicted projection verdict and its new wire field, an unknown high-water mark no longer reading as zero, the SQL allow-list resolving schemas (a bare name that would fall through search_path is now refused), and portable MCP tool schemas. The last two are behavioural — see Upgrade Notes.

Upgrading from 1.1.0-beta.2

beta.3 fixes three things a large multi-store fleet runs into immediately, all of them cases where a screen stated something it could not support:

  • On a service with several event stores, the Event Store Explorer opened on whichever store the host happened to register first while the picker read "All stores". On one fleet that was an ancillary authorization store, so the Configuration tab described a 514-database service as having three event types and one projection. It now opens on the store the service calls its main one, and the picker names that store instead of claiming a scope the tab does not have.
  • The Streams tab could load forever on a store with hundreds of databases — no timeout, no error, and the Refresh button disabled for the whole wait, so the one control that could retry was the one control you could not reach. Every Event Store Explorer read now has a 45-second deadline that turns a hang into an error you can retry, and Refresh stays clickable while a read is in flight.
  • The rollout panel could fill the top of the screen with an answer it did not have — "Rollout in progress" over a dozen identical rows reading "Cannot tell", on a store where no shard can be measured at all. It now collapses to a single line saying so, and carries the age of the snapshot it is describing, because "in progress" is a present-tense claim.

(No schema or data changes, so upgrading from beta.2 is a package bump.)

Upgrading from 1.1.0-beta.1

beta.2 is worth taking if you run a large fleet, and especially if your console has ever looked calm while something was wrong. Most of Every screen says how old it is below landed after beta.1, including the one that matters most: a console that has stopped applying telemetry now says so instead of reporting healthy.

(beta.1 was published as 1.1.0-beta.1. An earlier draft of this page spelled it 1.1.0-beta-1, with a hyphen, which is not a version that exists on NuGet.)

Projections at fleet scale ​

The Operations screen was designed against a test fleet of two databases. A production console running 512 databases and 11,637 shards met it very differently, and much of this release is that gap.

  • The shard is the row now, and the screen opens on the exceptions instead of on a complete inventory you have to search.
  • Shard progression is queried server-side — paged and filtered, with lag derived where the rows already live. Connect used to ship every shard state to every browser tab: 13.7 MB, rebuilt client-side into three indexes costing 148.9 MB of heap. The live update was a broadcast too, so a dashboard tab held ~37 MB of shard state it never rendered.
  • The Operations tab no longer blocks the browser. It spent 42–63 seconds in a single task on a 512-database store, because every derived lag reading re-walked all 11,637 shard states eight to ten times per row.
  • Four projection-health signals were sampling one shard out of 512. A paused shard in a sharded fleet rendered "Healthy · Up to date".
  • Alerts no longer go quiet exactly where they matter. A shard whose lag could not be computed was filtered out of the roll-up and raised nothing at all, and a shard that was never assigned an agent raised nothing because the check returned early on a null heartbeat. Relatedly, a projection whose lag cannot be measured no longer renders as "Healthy · Up to date" — twenty of twenty read that way while the Max Lag tile simultaneously called their gap unmeasurable.
  • Per-tenant metric roll-ups are no longer broadcast to every tab every 30 seconds (2,173 rows per tick on one field fleet); that answer is requested on demand.
  • Agent state is no longer republished when nothing changed — 2.76 MB of byte-identical state every 30 seconds, plus 145 heartbeat messages per tick.
  • Dead-letter summarize stopped fanning out one message per message database — 518 messages every 30 seconds on every route, which drowned the 500-entry Message Log at 99.4% occupancy.
  • A shard that two processes both believe they own is Conflicted, not Paused. A ProgressionProgressOutOfOrderException stops a shard, and the store will never restart it on its own. It used to render as Paused — the one verdict that sends an operator to Restart, which cannot work while the second owner is still writing. The new verdict ranks above Failing and is answered by finding the other owner. It rides a new wire field; older satellites fall back to the exception type and reach the same verdict, so nothing degrades in the meantime.
  • An unknown high-water mark says so. A store with per-tenant partitioning has no store-global mark, and the Operations header used to render that as "High-Water Mark 0 — no events appended yet" directly above a grid showing shards at 4,403 / 1,109 / 70. Unknown is now reported as unknown, with the reason.

Restart a stuck high water loop ​

Every async daemon runs a high water detection loop, and when it stops polling, every projection on that database stops seeing new events together while each one still looks healthy on its own. Each node now reports its own loops, because the loop is per node: under Wolverine-managed distribution, every node running an agent for a database runs its own. Expand a store's High Water Mark row on the Projections page to see them. A loop its node reports stale gets a Restart detection button, and the row carries a Loop stale tag so you don't have to open every row to find one.

The restart never moves the high water mark, so it cannot skip an event. It is acknowledged only once the loop has polled again. If the database cannot answer, the restart leaves the loop stopped, and the outcome says so rather than reporting success. It needs the new projection.restart-high-water capability, is audited, and is available to agents as list_high_water_loops / restart_high_water_detection. Multi-database stores whose projections Wolverine distributes are read too, since Wolverine 6.51.2 (wolverine#5092). See High Water Detection.

Rewind every tenant of a partitioned store ​

On a store that partitions events per tenant, Rewind on a subscription's row now rewinds every tenant. There is no store-global agent on such a store, so this used to fail with "No live subscription agent". The monitored service fans the rewind out to each tenant agent the cluster runs and forwards each one to its owning node. The row shows k of N until every tenant has reported, and each tenant gets its own timeline entry. A tenant's row can be rewound on its own, to a sequence included. To Sequence without a tenant is refused in the dialog, by the service, and when a rewind is scheduled, because a sequence is only a position within one tenant's events. Per-tenant rewinds need JasperFx 2.91 and Marten 9.51 on the monitored service. An older stack says so instead of returning the daemon's raw NotSupportedException. See Projections.

The Event Model is a live screen ​

The Event Model is now assembled from what your fleet actually does, rather than from a diagram someone drew.

  • One semantic model, shared with the rest of the Critter Stack rather than rebuilt inside the console, merged from four sources — including what Wolverine observes at runtime.
  • It renders fleet-wide with no traced subject. Previously the Workflow screen's Event Model mode showed nothing until you traced something, even though the merged model needs no subject.
  • It refreshes when the model changes instead of being fetched once on mount and then quietly going stale as services connect.
  • Where sources disagree, the screen says so rather than silently preferring one.
  • A service that hosts several Event Models keeps all of them. Each of the three stores can name its own model, so a modular monolith with a store per module legitimately assembles several — and the console used to pick one name and fold the rest into it, losing a model's name outright with nothing reporting it. A service now reports the set, and where a surface can still carry only one model the fold is explicit and says so on the canvas, naming the models that went into it.
  • An observed slice names the service it was watched on — critterwatch://observed/{service}, and every service when two contribute to one slice — so a fleet-level disagreement names its sources and not only its rungs.
  • A projection's apply set reaches the canvas from the code, not only from a running store. The source generator now reads a projection class's Apply / When / Create / ShouldDelete signatures and contributes a View slice named after the document, carrying the events it consumes. It is named the way the store's own registry-derived slice is, so the two merge into one slice — and where the static reading and the running store disagree about what a projection applies, that is a hotspot on the canvas naming both, which is a finding rather than a conflict.
  • A Bobcat on a newer JasperFx than the console no longer empties the evidence. One vocabulary member the console had never heard of used to discard the whole spec-evidence fetch — the run and every scenario already retrieved. Unknown members now read as unrecognised, and a model that cannot be read keeps the run beside a note saying so.
  • Drift is shown from evidence, not from a declaration — specification runs are read over HTTP, so a slice's colour reflects whether its behaviour is actually covered and passing.

Schedule a rebuild for tonight ​

  • "Rebuild later…" on a projection row, defaulting to 03:00 tomorrow — the point of the feature is to move a rebuild out of business hours, so the default already is.
  • The schedule lives in the monitored service's own message store, which means it appears on the Scheduled Messages page you already have, and can be cancelled or moved there. A console-held schedule would have been invisible and uncancellable.
  • Acceptance is acknowledged immediately, carrying the time it will run, and the real outcome is acknowledged when it runs. A row no longer sits "pending" for hours against a rebuild that is correctly waiting for 03:00.
  • Rebuild every tenant of a projection from one place. On a per-tenant partitioned store each tenant owns its own shard, so a store-global rebuild would have left every tenant partition untouched — the console now fans out per tenant and names the count before it does, because at fleet scale that is hundreds of simultaneous rebuilds.
  • Scheduling, cancelling and rescheduling are available to AI agents through licensed MCP tools.
  • A subscription rewind can be scheduled the same way. The Rewind dialog gains Now / Later, with every mode — to the beginning, a sequence or a timestamp. It lives in the same message store, shows on the same Scheduled Messages page, and a malformed target is refused when you schedule it rather than at 03:00. Requires the monitored service to be on this release of Wolverine.CritterWatch; an older one has no handler for it and says so, rather than rewinding immediately.

WARNING

A schedule is only as durable as the queue it sits on. A service whose local queue is the default in-memory one loses a 03:00 rebuild to a 23:00 redeploy, silently. CritterWatch refuses a schedule it cannot honour rather than accepting one it will drop.

Schedules that repeat ​

Wolverine 6.34 can run a message on a cron schedule, and CritterWatch is the console for it.

  • Register a schedule in one line — opts.Schedules.ScheduleRecurring<NightlyRollup>("0 2 * * *"). Registration is the opt-in; there is no separate feature flag.
  • A fleet-wide Schedule Explorer lists every recurring schedule across every monitored service — name, cron expression, time zone, the message type each occurrence publishes, and when it next fires. Scoped to one service with the header selector, like the other explorers.
  • Pause and resume a schedule from the console. Pausing stops it producing occurrences and cancels the one already pre-scheduled, so nothing fires in the gap. Both are separate permissions (recurring-schedule.pause / .resume), because stopping production work and restoring it are not equally consequential.
  • The occurrences themselves were always visible. A pre-scheduled firing is an ordinary durable inbox row, so it appears on the Scheduled Messages page you already have. What was missing was the schedule — what fires, on what cadence, and when next.
  • Available to AI agents through list_recurring_schedules, pause_recurring_schedule and resume_recurring_schedule.

WARNING

A pause is only durable if the service can store it. A service whose message store carries the recurring tracking table records the pause as a row that survives a restart. One without it applies the pause to the running agent's memory instead — real, but lost on the next restart. The console reports the difference rather than showing both as "paused", because a pause that a redeploy will quietly undo is worse than one that visibly failed.

TIP

Resuming does not backfill. Occurrences that would have fired while a schedule was paused are gone by design — the next occurrence is computed strictly after the resume. A missed firing stays a missed firing, which is the whole reason the schedule is owned by an agent rather than by a message that reschedules itself: one lost link cannot end the chain.

TIP

Schedules live on a service's main database and are never per-tenant, even on a multi-tenant service. The next-fire time and paused flag shown in the console are as of that service's last capability announcement; a pause or resume you perform from the console updates the row immediately from the acknowledgement.

Compact old streams on a schedule ​

Event streams grow forever. Compaction folds a stream's history into a single snapshot event — destructive and irreversible, but invisible from the read side, because the stream still aggregates to the same state and keeps its version. CritterWatch could already do that to one stream, by hand, from a button on the Event Store Explorer. That is not an answer for a fleet.

A compaction policy says it declaratively: compact any stream of this aggregate type whose un-compacted growth exceeds this many events, on this cron schedule.

csharp
monitoring.CompactStreams(p =>
{
    p.Name = "BoardHistory";
    p.AggregateType = typeof(Board);
    p.GrowthThreshold = 1_000;
    p.Cron = "0 2 * * *";
});
  • A policy is a recurring schedule, so everything the section above describes applies to it unchanged: one tick per cluster rather than one per node, a Schedule Explorer row, and pause and resume from the console under the same permissions.
  • Growth is measured since the last compaction, not from zero. A threshold on the raw version would re-fire forever on a stream that has already been compacted; a threshold on growth reads zero on one that is fully compacted, which is what an operator means by "this stream is fine".
  • Every run is reported on the Timeline — streams matched, compacted, skipped and failed, and why a run could not happen. streamsMatched is the whole backlog rather than the tick's share, so a policy working through a large store over successive nights is visible as progress.
  • Run one now from the Schedule Explorer, or over MCP, instead of waiting for the cron time.
  • Available to AI agents through list_compaction_policies, get_compaction_activity, run_compaction_policy, and pause and resume — the last three permission-gated, and the read tools explicit that a dry run changed nothing.

WARNING

A new policy is a dry run until you arm it, and arming it is a code change. A dry-run policy reports exactly what it would compact and deletes nothing. Arming is deliberately not an operator action: the console can stop a policy, run one, and show what one did, but it cannot turn a reporting policy into a destructive one — that stays in the code you deploy and review. The intended loop is declare, deploy, run it once from the console, read the report, then decide.

TIP

All three event stores compact — Marten, Polecat and Fisher — so a policy behaves the same whichever one a service runs on. Polecat needed 5.25.0 to get there: it implemented the typed compaction path but not the untyped one a policy needs, so it selected the right streams and then refused every one of them (polecat#572). Against an older Polecat a policy still reports that, once per run as a store-capability error rather than as a failure per stream.

TIP

Declaring your first policy creates a table. A policy is driven by a recurring schedule, and registering the first schedule is how a service opts in to the recurring-message agent — which provisions its tracking table on the service's main database. A service that declares no policy gets no schedule, no agent and no schema change, so adding the package alone never touches your database.

TIP

MaxStreamsPerTick caps how much one run does, and it exists for the first armed run against a store with a long backlog: the policy works through it over successive ticks rather than in one stampede. It matters most on SQLite, which takes a single writer per file.

Per-tenant subscription lag ​

Subscriptions are reported per tenant, not just per store. On a tenant-partitioned event store a subscription shard carries its own progression per tenant, and the console now reads and renders it — so a tenant that has stopped advancing is visible instead of averaged away.

TIP

A tenant with no progression row is a different thing from a tenant that is fully behind, and the console now distinguishes them. A missing row usually means the tenant has no registered partition, or a shard that started and never wrote — both are diagnosable, and neither is "caught up".

Sharper questions of the event store ​

The Event Store Explorer's by-metadata search and the query_events tool both gain the filters the underlying store query grew this cycle, and they AND together with the facets already there:

  • Several event types at once, as a union rather than one query per type.
  • A timestamp window and a sequence window, each inclusive and each valid half-open.
  • Counting over a window — how many of this event between Tuesday and Thursday is now one question rather than a page-by-page tally.
  • DCB tags combine with all of it. The Explorer's separate By Tag mode is gone: tags are one facet of the By Filters view, AND-ed with every other filter, with a total count of the combined match. query_events does the same.
  • Tag queries work on every store. The old By Tag mode used a tag query that Fisher (SQLite) does not implement and Polecat (SQL Server) refuses store-wide, so on two of the three stores it answered with an error. The combined query is honoured by all three.
  • By-Metadata results show their payloads. Every row used to render an empty body — the rows arrived in a different shape from the one the timeline reads. Fixed in the console, so it holds even against monitored services that have not upgraded.
  • The Explorer opt-in now covers every event read. EnableEventStoreExplorer used to gate the stream and tag views but not the by-metadata query or query_events, which return full event bodies. It gates all of them now — a behaviour change for services that have not opted in; see the upgrade notes.
  • Listings answer once per database on a multi-database store. On a database-per-tenant or sharded store the Streams tab used to show whichever database the store's default session resolved, returned as though it were everything — 512 databases, one database's rows, nothing saying so. Recent streams and projection statuses now fan out over the store's databases and every row says which one it came from; a row opened from the listing reads its stream from that database, since a stream id is unique within a database and not across them. A single-database store keeps its one query. Needs a current client package on the monitored service. All three stores answer per database: SQL Server and SQLite always did, and PostgreSQL does from Marten 9.37 (marten#5383). The version that matters is the monitored service's own, not the console's — a service still on Marten 9.36 or earlier refuses the per-database read by name, per database, which remains an honest empty rather than one database's rows dressed as the whole store.
  • A projection's daemon state is what the store actually knows. The three stores had each decided independently what a shard's state field meant; upstream now rules it a fact about the running daemon, with Unknown meaning "no daemon to ask" — which the console keeps distinct from Stopped, the reading an operator acts on and the one SQL Server used to report for it. Lag on a stopped daemon is real again too: the store head is the store head, not where the daemon got to, so a stopped projection no longer reads as zero behind at the moment it matters.

An offset-less timestamp is read as UTC, not as the console host's local zone. Applying a server's zone would shift a window by hours without saying so, and make two consoles disagree about the same store.

WARNING

The results you are shown are the results you asked for, or you are told they are not. A monitored service running an older client package does not understand a filter it has never heard of — it drops it while deserializing the request, so the filter never reaches the store to be refused, and the store's own guard against silently-ignored filters cannot see it. Unfiltered results would come back looking exactly like filtered ones.

So a service now echoes back the filters it actually applied, and both the console and the tool compare that against what was sent. If anything was dropped, the filters are named and no rows are rendered at all — an unfiltered page is worse than no page, because it reads as an answer. A service that echoes nothing is treated as having dropped everything, since the absence of an answer is not evidence that the answer was yes.

TIP

Tag queries combine with the filters above. A DCB tag match ANDs with event types, time windows and the rest, so "events tagged customer: acme, of this type, between Tuesday and Thursday" is a single call rather than two queries intersected by hand.

This landed late. The composable query object had no name/value tag member at all — the typed form resolves tag types against the store's own registered graph, which a console cannot reach, so a dictionary tag query could only go through a separate method that accepts no other filter. That gap is jasperfx#801, and closing it took all three event stores.

⚠️ A tags-only query against a monitored service too old to honour tag filters on the combined path has no totalCount. It is detected — its applied-filter echo lacks TagValues — and re-asked on the older standalone path, which that service does speak. Every other query, tags-only included, answers with a real total. Do not read a missing totalCount as zero.

AI and MCP ​

The CritterWatch.Mcp server grows from 48 tools in 1.0 to 83 in 1.1: 37 read tools and 46 licensed, capability-gated actions.

  • Query the event store in SQL — the Event Explorer's new SQL view and the query_sql read tool run one read-only SELECTon the monitored service, on its own connection, so the console never holds database credentials and a Marten, Polecat or Fisher service is reached the same way. The statement is allow-listed to the tables the store declares (a reference outside them is refused with the list), runs inside a read-only transaction that is always rolled back, is statement-timeout bounded and row-capped, and says when the cap truncated the answer rather than letting a floor read as a total. It is the escape hatch beside the structured query for the join, the GROUP BY and the per-stream count the filters cannot express. It carries its own capability (sql.query in the console, mcp.sql.query over MCP) and every run is written to the audit log — the outcome as well as the request, so a refused statement no longer reads as "Ran SQL". Opt-in on the monitored service: EnableSqlQuery, which follows EnableEventStoreExplorer unless set. ⚠️ The allow-list resolves schemas. An earlier build matched bare table names and ignored the schema, so one service's SQL view could read another service's event store in the same database. A bare reference is now qualified against the store's own schema and matched qualified-only; a reference that would have resolved elsewhere through search_path is refused.

  • Step a projection through its events as a read tool, alongside a diagnostic that explains what a projection did with a stream.

  • Manage stream compaction policies — list what is declared, read what each run did, and run, pause or resume one. The read tools state plainly whether a run was a dry run that changed nothing, so an agent cannot report compaction that did not happen.

  • Every tool remains licensed and capability-gated. The stepper and diagnostic only read. The compaction run, pause and resume tools act on the service: running a policy needs compaction-policy.run, and pausing or resuming one needs the same capabilities as pausing or resuming a recurring schedule.

  • Every tool's input schema is portable. Every tool advertised schemas using type unions with "null", the format keyword and no additionalProperties — the constructs MCP Inspector, OpenAI strict function-calling and Gemini's schema subset flag or reject. They now emit the portable subset, which is the difference between the MCP surface working with a given client and not.

  • Every tool says whether it only reads and whether it can destroy something. All 83 tools declare the MCP readOnlyHint and destructiveHint annotations explicitly, so an agent client can ask the user before a destructive call instead of prompting for everything or nothing. The classification is conservative: every rebuild, every replay and every injected fault is destructive. See tool annotations.

  • A read-only switch. AddCritterWatchMcp(o => o.ReadOnly = true), or CritterWatch:Mcp:ReadOnly = true in configuration, or WithCritterWatchMcp(readOnly: true) from Aspire, leaves only the 37 read tools registered, so an agent can investigate and cannot act. Action tools are absent from tools/list, a backstop refuses any CritterWatch action reached another way with the deny code McpReadOnly, and the server tells the agent it is read-only. Reads stay licence- and capability-gated. Off by default. See read-only.

  • The audit log says how each action arrived. Every row now carries a channel beside its user: Console, MCP (with the client the request named) or System (the console recording something itself, such as a service's answer). The Audit Log page has a Channel column and filter, and the endpoint takes ?channel=. Rows from before 1.1 read Unknown and are never guessed. An MCP action is recognized by its relay ticket, which a monitored service cannot forge, and reads as MCP even with RBAC off. Deriving the channel also fixed attribution: an alert acknowledge, snooze or clear, a metrics delete, an eviction or a license-slot change made through MCP was recorded as anonymous, and now names its caller. The high water mark advance's outcome row is now system, as other outcomes are. See Audit Log › Channels.

  • The server asks before an irreversible tool runs. Thirteen tools whose calls cannot be undone now answer their first call with a preview of what they would do and a single-use confirmationToken, bound to the caller, the tool and the arguments. Nothing changes until the same call comes back with the token. Covered: discarding dead letters, discarding a dead-letter failure group, updating a document, a permanent document delete, removing or hard-deleting a tenant, ejecting a projection or a retired version, cancelling a scheduled message, deleting metrics, evicting a service, running an armed compaction policy, and triggering the schedule of one. Licence and RBAC are checked on both calls. A client set to auto-approve can still chain the two calls; the guarantee is that the agent sees the consequence first and the call that runs is the one previewed. See confirmation tokens.

  • Conversations for agents. list_conversations lists recent conversations, narrowed by message type, service, failures only or a start time. get_conversation opens one by its conversation id or a correlation id, so a correlation id from a failing test or a log line goes straight to the conversation behind it. It returns the hops, the Why graph's edges marked observed or inferred, and the When view's timing and gaps, derived by the same code the screen uses. completeness says when hops may be missing, and a message that failed and then succeeded is recovered, not failed. cw-conversations prints the same JSON from a shell. Both need the new mcp.conversations.read capability, because a failed hop's status description is exception text. See Conversations.

Alert triage by a language model ​

Alerts that fire together are already coalesced into one batch. That batch can now be handed to a model, which writes a short assessment of what is probably going on and suggests what to do about it. The narrative is stored on each alert it names and rendered beside it in the console.

Off unless you turn it on. It calls a paid model on a schedule you do not control, and an alert storm is when it fires hardest — so it is a deliberate opt-in, the model client is yours to register (any Microsoft.Extensions.AI provider; CritterWatch takes no vendor dependency), and the token budget is enforced by a middleware that dead-letters an over-budget request rather than paying for it.

  • One request per batch, never one per alert. The correlation is the feature: the model is shown the alerts that fired together as one possible incident, which is what an operator otherwise has to do in their head across five notifications.
  • Suggestions mark the buttons the console already offers; they never add new ones. A suggested action lights up the existing remediation button for that alert type, and pressing it goes through the ordinary command path with the ordinary permission check.
  • Nothing it returns is executed, and nothing it says grants anything. The narrative is labelled as model output on its face, and no alert is raised, resolved or suppressed because of it.
  • Because a model call is an ordinary message on a queue, the console monitors it like any other: latency, failures, retries and dead letters all appear on the screens you already use.

Operator commands are permission-checked in the console, not just over MCP ​

If your host registers an authorizer, the console now enforces capabilities on every operator command — dead-letter replay and discard, listener pause/restart/drain, projection and agent operations, scheduled-message edits. Previously several of these were checked for an AI agent over MCP and not for a person clicking the same button, which is the opposite of what the capability list implies.

⚠️ If you have real permissions wired, read the upgrade notes — an operator who could run one of these before may now be denied until the capability is assigned. Hosts with no authorizer registered are unaffected: the default allows everything.

RBAC: the MCP and relay gates agree, and MCP actions work on an enforced host ​

On a console that registers a real ICritterWatchAuthorizer, every MCP action tool reported accepted: true and did nothing. The tool authorized the caller and then published a command carrying no principal; the relay authorizes again, is fail-closed on a missing principal, and denied — without ever asking your authorizer, so no grant could have helped. The refusal had nowhere to go, because the caller was not a websocket, so the only trace was a log line. Hosts that never registered an authorizer were unaffected, which is why this went unnoticed.

The caller's identity now travels with the command, in this process, so the relay decides about the person who actually asked.

⚠️ Two capability families now name a different resource — tenant and projection commands. If your authorizer decides using the resource argument, see Upgrade Notes before upgrading.

Run the console inside your own application ​

CritterWatch no longer has to be a separate deployment. Two calls mount it inside a host you already run, monitoring that host.

csharp
opts.AddCritterWatchEmbedded(postgresSource);   // inside your own UseWolverine — an NpgsqlDataSource on Marten,
                                                // a connection string on Polecat and Fisher
app.UseCritterWatchEmbedded();                  // mounts the UI under /critterwatch
  • No new package and no new project. Embedded is a registration mode on the three packages that already exist — CritterWatch (Marten/PostgreSQL), CritterWatch.SqlServer (Polecat) and CritterWatch.Sqlite (Fisher).
  • The console's own data lives in its own store, isolated from yours by construction: it needs string stream identity where your application almost certainly uses Guid, and that is a per-store setting.
  • Your routes are untouched. The UI mounts under a sub-path rather than taking the route table, so a host route that returns 404 today still does.
  • The console gets its own wolverine_* envelope tables — one runtime, two message stores. They are shared with the host only when the host has set opts.Durability.MessageStorageSchemaName (⚠️ not the same-named property inside IntegrateWithWolverine(o => ...)). A maintenance routine that rebuilds only the main message store will not touch them — see Embedded for the loop that covers every store.
  • The host is monitored automatically. Embedded has no separate service to instrument, so the console registers the monitored half on the host's behalf; telemetry travels in-process.
  • Nothing to compile. The console's handlers and HTTP endpoints ship pre-generated and are registered directly, so your host needs no WolverineFx.RuntimeCompilation on the console's account and your own codegen write covers only your handlers.
  • Behind your own authorization. UseCritterWatchEmbedded(configureEndpoints: ...) applies a policy to every endpoint the mount creates (the UI, the HTTP API and the SignalR hub) and to nothing of yours. ConfigureMonitoring reaches the explorer switches a standalone service would set.
  • Your application's behaviour stays yours. The console's retry ladders, message grouping and durability settings apply to its own handlers and queues only, its HTTP API speaks its own JSON without touching your JsonOptions, and your Wolverine tracked sessions (and Bobcat specs) ignore its traffic. See Embedded for the full list, including the few host-wide registrations it does make.

WARNING

Single instance only. Embedded self-monitoring rides in-process local:// messaging, which cannot cross nodes — so N nodes means N consoles writing the same streams, and no partitioning topology fixes it. The console detects and reports this at runtime. Use the standalone console for a clustered deployment.

Every screen says how old it is ​

A console watching 514 databases and ~2,100 tenants rendered a 37-day-old snapshot as current, and nothing on any screen said so. That is the worst failure this product can have — an operator looking at a calm dashboard and concluding the fleet is calm — and most of this section is that gap.

  • A console that has stopped applying telemetry now says so. Its self-ingest health check was attached to transport listeners only, and the queue where telemetry is applied is a local one, so the check never saw its failures: in a reproduction it answered healthy over 849 constraint violations in 30 minutes. It now reports degraded within seconds, naming the exception.
  • Liveness alerts stand down against a frozen snapshot instead of measuring the console's own ingest gap as agent silence. One field console raised 25 Critical "agent down" alerts against agents last observed five weeks earlier. Alerts that read a snapshot's content still run, and a console with no ingest stamp is not treated as frozen, so upgrading does not silence anything on a healthy fleet.
  • An observation age rides the rows, tiles and summaries that previously showed a value with no indication of when it was true. The one freshness indicator the console had measured the browser's own connection, which is green precisely when a stalled console looks healthiest.
  • The landing page no longer states "No dead letter messages anywhere" over 15,972 of them for the first couple of seconds of every load, and the Operations tiles no longer claim "no events appended yet" over 2.66M events shown directly below them. Both were guards wired to a snapshot arrived rather than the counts arrived.
  • Two ways a dead projection shard rendered as calm are closed. "Idle" conflated awaiting its first event with no agent is running it, and a frozen high-water mark subtracted to a healthy gap. Relatedly, 7,775 never-advanced shards no longer raise a permanent unmeasurable-lag warning that the same console calls Idle.
  • The Timeline carries fleet events. It was 95.7% one flapping self-check — 180 of 188 entries over a month were the console watching itself, and no fleet event reached it at all.

Operating a fleet of this size ​

  • The Operations route stops walking every progression row on mount. It fetched all 12,814 — 6.6 MB in 26 requests, 23 seconds of blocked main thread — to render a 25-row grid that reads none of them. Its header tiles are computed server-side now.
  • The shard grid sorts and filters by health and lag, and the health chips above it are filters rather than labels. 467 pages with no sortable column is not a grid you can use.
  • Total backlog is a real number. Summing each shard's gap would have been wrong by 3.3x, because the high-water mark is per (database, tenant) — shards on one tenant are not independent measurements. The roll-up accounts for that.
  • A blue/green projection rollout is readable. The projection version is a dimension the server can group by rather than a substring; a shard row names the fleet that owns it; and each row carries a lag delta, so a rate and an ETA are measurable rather than eyeballed.
  • Per-tenant alerting works above 25 tenants. Two independent ceilings used to collapse 2,173 tenants into a single roll-up whose membership was a top-five list inside a sentence. Tenants now ride the alert as data, a named few can keep their own alert above the ceiling, and one tenant can be silenced without going blind to its projection.
  • An acknowledged alert is acknowledged everywhere, rather than staying acknowledged only in the tab that did it until every other console reloads.
  • Several MCP tools reported a clean fleet from an empty source. get_backlog_state and list_backlog_hotspots read persistence counts that were {} while dead-letter summarize found 15,993 on the same console; list_degraded_surfaces reported every endpoint as degraded by excluding a status the endpoint vocabulary never emits; get_projection_lag returned 11,656 unpaged rows with no tenant, store, gap or verdict.
  • "All stores" lists every store. An empty store identifier resolved to a single service, so the Streams tab could report six streams from an ancillary store as the whole fleet.
  • The dev fleet boots from a Windows checkout. Without a .gitattributes the tenant-database init script was checked out CRLF and the Postgres container exited 127.

A recurring schedule says when it last fired ​

Following the Schedules that repeat section above, and worth calling out because it changes what the Schedule Explorer's numbers mean.

  • Every runtime reading carries the instant it was true. A schedule's next-fire time and paused flag ride the capability announcement, which is only rewritten when the schedule's definition changes — so those readings were frozen at registration time and could sit hours stale with nothing saying so. Each service now reports its live schedule state on its own telemetry cadence, and the row says as of 40s ago for a live reading or announced 6h ago for one that is not. They are different claims and the screen words them differently.
  • "Is it still firing?" is answerable. A new Last handled column shows when a monitored node last actually ran an occurrence — an observed run, not the schedule's own bookkeeping. The two diverge exactly where it matters: a schedule whose message has no subscribers keeps a moving next-fire time and never handles anything. An empty column means not observed — an occurrence handled by a different service is invisible to it — and never idle.
  • A pause you performed survives a reload. The console keeps its own record of every pause or resume a service verified, so the row no longer reverts to Running until the next announcement.

Follow a replay or discard to the end ​

A replay, discard or edit on the Dead Letters page, or from an agent over MCP, is now an operation you can follow. The new Operations tab lists the recent ones, newest first:

  • Progress across every database it touched, for example "4 of 6 databases". Each service announces the databases it will run against, and each database answers when it is done.
  • How many dead letters it affected. With Wolverine 6.54 the relational stores report an exact count per database. A store that cannot count says not reported, never 0.
  • Where it failed, and why. A database whose replay or discard throws is listed with its error. Before this, the failure went into the monitored service's own error handling, and the console never heard of it.
  • Who ran it, from where, and why: the user, the channel (Console or MCP) and an optional reason, which is also written on the operation's audit row. The bulk confirmations ask for the reason, and replay_dead_letters / discard_dead_letters take one.
  • A message's replay history. The detail drawer and query_dead_letters say how many times a dead letter was replayed and failed again, from Wolverine 6.54's replay history.

An operation that nothing answers for ten minutes reads Unconfirmed, not failed. A service on a Wolverine.CritterWatch client older than 1.1 runs the operation but never reports an outcome, so the console can only say it was not told. See Dead Letters.

Dead letters as failure groups ​

A new Failure Groups tab on the Dead Letters page groups dead letters by the same failure, as Wolverine 6.54 fingerprints them (the exception and where it was thrown), or by message type, endpoint or exception type. The console reads every service and merges the groups itself, so one row covers every service and database the failure spans, with its count and first and last failure. Click a group to load its dead letters on the Messages tab, where they can be replayed, discarded or exported. A partial read is flagged, never shown as complete. Agents get the same view with the MCP tool summarize_dead_letter_groups, and query_dead_letters takes a failureGroup.

Failure groups need Wolverine.CritterWatch 1.1.0 on the service. The 1.1 release candidates, 1.1.0-rc.3 included, do not advertise the feature, so the console treats them as 1.0. A service on an older client keeps working: the console never sends it the new request, which it does not implement, and groups its dead letters by exception type instead. See Dead Letters › Failure Groups.

Each group can be replayed, archived or discarded as a whole, with a required reason, and archived groups restored. Archive takes a failure out of the queue and its counts without losing it; it needs the new dlq.archive capability. One operation runs per group at a time, a running one can be cancelled (the databases it has not started yet are skipped), and a finished replay can say how many of its dead letters failed again. A note on a group, for the next operator, stays with it through replays (dlq.note). Agents get replay_dead_letter_group, archive_dead_letter_group, restore_dead_letter_group and discard_dead_letter_group.

Check for updates ​

The About widget tells you when a newer CritterWatch is on NuGet, and links the release notes for the version you are running.

Fixed ​

Slack notifications never fired from a standalone console. Slack's handlers were discovered by assembly scan, which a standalone console's pre-generated handler registry never includes, so no alert ever reached Slack. Slack now plugs into a notification hook the console calls with every alert batch, off the alert path, and the package no longer requires the Marten console package: it works with CritterWatch, CritterWatch.SqlServer and CritterWatch.Sqlite.

The broker-less HTTP channel never worked as documented. A service with http(s) CritterWatch URIs did not start (Unknown Transport scheme 'http': Wolverine registers its HTTP transport only for ToHttpEndpoint()), and once it did, the console's /_wolverine/invoke rejected every telemetry envelope, because that route reads CritterWatch's compressed wire format as plain JSON. The channel now uses Wolverine's /_wolverine/batch/{queue} route on both sides: the console declares its critterwatch ingress queue and registers the HTTP transport itself, and a service adds one line. A misconfigured URI stops the service at startup with the fix in the message. The HTTP and gRPC channels are both covered end to end now. Change your URIs: see Upgrade Notes.

Alert Configuration's History tab stayed empty. Saving a threshold never recorded a change; it does now.

A store whose projection daemon Wolverine does not manage reported no live projection state. A Solo or HotCold store without managed distribution, and every Fisher store, showed each projection as it was at startup. A projection that ran and caught up still read idle, and raised a Critical "nothing is advancing it" alert. The console now subscribes to those stores' shard state itself.

Projection staleness was never measured. A projection's last-advanced time was lost on the bare state that follows each advance, so no projection on a fresh fleet had one. It is kept now, and a caught-up projection is never called stale however long ago it last moved.

An edit to a projected document was not checked for a concurrent change. A Marten single-stream projection's document keeps its stream's version, so an edit leaves that version unchanged, and a second operator holding the same version could overwrite the first edit silently. The version token now includes the document's last-modified time.

A compaction policy's schedule said it had never run. Its detail page did not load the policy's recent runs.

The update check compared a source build against the wrong version. A console built from source now reports the version after the latest tag rather than a placeholder.

The Event Model named Wolverine's generated handler class. A slice whose handler publishes a message showed StartTripHandler939323516 rather than StartTripHandler, and marked the slice as a disagreement between sources. It shows the handler you wrote; for services that have not upgraded too. The upstream fix is wolverine#5224.

Most of the Event Model's "sources disagree" marks were not disagreements. A slice seen in production and also derived from the code was marked as a disagreement about where it came from: 13 of 38 marks on a 32-service fleet. Those are gone (JasperFx 2.96, jasperfx#1083). A genuine disagreement is now a quiet ≠ on the card whose role is disputed, with both claims in its tooltip, rather than a red sticky on the canvas (@jasperfx/event-model-vue 0.16, jasperfx#1084), and the slice strip marks each such slice with a ≠ too, with a count beside its label.

The Event Model missed aggregates on Fisher, and those loaded with [WriteModel] / [ReadModel]. The source generator that reads a service's handlers recognised only the Marten and Polecat aggregate attributes. It now reads Fisher's ([WriteAggregate], [ReadAggregate], [ConsistentAggregate], [BoundaryModel], [AggregateHandler]) and Wolverine's store-agnostic [WriteModel] / [ReadModel] too.

An empty Event Model said a service had not pushed a manifest when it had. A service with nothing to model (documents only, or publish only) is now described as that.

A message that failed and then succeeded still read as failed. In Conversations, a message retried or replayed into success stayed red and still offered Replay and Discard for a message no longer in the dead-letter queue. It is now marked ↻ as recovered, and the list's failed tag counts only messages that are still failed.

Conversations opened from the Workflow screen landed on the list. Opening one from a Workflow edge now shows the conversation. An edge into an event lists the conversations through the message whose handler appended it, rather than none, and the flow graph no longer draws a second copy of a handler whose append only the runtime saw.

Save & Replay, and fourteen other operator actions, did nothing when clicked. A confirmation popover had stopped binding to its button, so the click opened nothing: no confirmation, no error, no command. Edit & Replay's Save & Replay was one of them. The others were across Service Details, Node Detail and Scheduled Messages; a fifteenth, Clear Node History, failed the same way through a dropdown. All fifteen are fixed, tests now click the real buttons, and a source check fails on the pattern that broke them.

Alert Config did not save the metrics thresholds. The metrics defaults and a service's overrides were edited under names the server does not use. The page showed its starting values instead of the stored ones, and a save kept the previous values: an edit to the warning or critical percentages or the failure-rate window was silently lost. Both are read and saved under the server's names now, and the fields the form does not edit (baseline days, the DLQ rates, hysteresis passes) are kept as they were. Choosing a service now shows that service's stored thresholds, and saving its message-type thresholds no longer resets its own per-service thresholds (nor the reverse). Per-message-type thresholds are saved, but 1.1 does not evaluate them yet: metrics alerts use the global and per-service values. Evaluating them is planned for 1.2.

A command sent through the console's own HTTP API was audited as channel Unknown, and refused on an RBAC-enforced host. The endpoints that send an operator command to a service (chaos monkey, tenants, projection administration, agent thresholds) authorized the caller, then handed the command on with nothing saying who sent it. So the audit row read Unknown, and with an ICritterWatchAuthorizer registered the relay refused the command for want of a caller, with no error returned to the HTTP client. Those commands now carry the request's caller, held in the console's memory, never on the wire. They are audited as Console and authorized as that caller.

A Datadog account on the EU1 site could not be reached. Every Site other than us1 became https://api.{Site}.datadoghq.com, so eu1 gave api.eu1.datadoghq.com, while EU1's API is api.datadoghq.eu. Both the trace provider and the metrics data source now map each site to its documented host (us1, us3, us5, eu1, ap1, ap2, us1-fed), and an unknown site fails at startup with the list instead of guessing a host. See Datadog Tracing.

AddCritterWatch(connectionString) did not cap the console's connection pool. Only this repository's development console applied the bounded default of 50, so a console built from the packages got Npgsql's 100 per process, and the CritterWatch:Postgres:MaxPoolSize key did nothing. That overload now applies 50, or the key's value. A Maximum Pool Size in the connection string still wins, and Npgsql's own spelling of it is now recognised: before, only MaxPoolSize was, so Maximum Pool Size=12 was silently replaced by the default. See Configuration.

The Rebuilds page was filtered by a service selector it does not show. Rebuilds is a fleet-wide page, so the header hides the service selector there, but the list still filtered by the service last chosen on another page. It lists every service's batches now.

Cancel All Matching on Scheduled Messages cancelled only the rows loaded on the page, not every match. It now cancels every scheduled message that matches.

An agent's dead-letter replay or discard was refused on any host with an authorizer, while the tool answered accepted: true. replay_dead_letters and discard_dead_letters published their command without the MCP relay ticket. So the console's relay saw no caller: with an ICritterWatchAuthorizer registered it refused the operation, and without one the audit row recorded channel Unknown instead of MCP. acknowledge_alert, snooze_alert, clear_alert, evict_service and delete_metrics_samples had the same omission. Their commands are not permission-checked at the relay, but they were audited as Unknown too. All seven now carry the ticket, and a test fails on any MCP action tool that publishes without it. Found by the 1.1 walkthrough of dead-letter management.

CritterWatch:Ai was never read, so switching alert triage off in configuration did nothing. The triage options were documented as bound from CritterWatch:Ai, and nothing bound them: a CritterWatch:Ai:Enabled = false in appsettings.json or an environment variable was silently ignored, and triage kept calling the model through the alert storm it was meant to be stopped for. When the host calls AddCritterWatchAlertTriage, CritterWatch:Ai now sets Enabled, Severities, Model and MaximumAlertsPerBatch, applied after the code so the configured value wins. Pass the configuration, AddCritterWatchAlertTriage(builder.Configuration, ...), and it sets the cost controls too (MaximumParallelCallouts and the Budget keys). Configuration alone still does not turn triage on. See LLM alert triage.

Dead-letter replays, discards and edits were never audited. The console's and MCP's dead-letter actions travel as one operation that bypassed the audit the other operator commands get, so a fleet-wide "Discard All Matching" left no trace. Each is now recorded as ReplayDlq, DiscardDlq or EditAndReplayDlq, one row per service it was sent to, with who, how (Console or MCP) and exactly which messages. An edit records the body's length, never the body. See Audit Log.

The Dead Letters page could act on a different set than its confirmation counted. A summary row's Replay All / Discard All hit every endpoint with that message and exception type rather than the row's own; the confirmation quoted all-time counts while the action used the selected range; the exception-message filter was a substring in the list and exact on the server; a tenant filter never left the browser, so "All Matching" acted on every tenant; and "All Matching" fanned out over the browser's own service list. Each action now sends the query its count came from: the summaries follow the range and endpoint, the exception filter is exact (and labelled so), "All Matching" is off with a reason while a filter the counts cannot see is set, and fleet-wide actions are fanned out by the console. Selected messages, and an Edit & Replay, now go only to the service that holds them.

Every endpoint's full settings reached the console, so the connect snapshot was about three times its size. Each endpoint's settings and their nested settings are meant to stay on the monitored service until an operator opens that endpoint. They were stripped from one of the two copies of the endpoint list the service sends at startup, and the console kept the other. On a 32-service fleet that was 70% of the connect snapshot every browser tab downloads on connect and reconnect. Both halves are fixed. The console no longer replaces a stripped endpoint with an unstripped one, and existing consoles heal themselves: each service's stored capabilities are rewritten once, on the first telemetry batch after the upgrade, with no redeploy needed. Measured on that fleet, the connect snapshot went from 10.8 MB to 3.5 MB and GET /api/critterwatch/services from 11.2 MB to 3.7 MB. The console fix and the heal work against existing monitored services; the 1.1 client package also shrinks what each service sends at startup.

"All stores" showed one store on a multi-store service. The Event Store Explorer's Configuration tab read the first event store in the service's capability document, which is host-registration order and not the main store — on a 514-database service that meant reporting three event types and one projection from an ancillary authorization store, with nothing on screen naming what had been read. The default is now the store the capability document identifies as the main one, and the picker displays its name. (The matching server-side read was fixed earlier in this release; this is the console half of the same assumption.)

An Event Store Explorer read that never came back left a spinner, not an error. These reads travel over the console's WebSocket, so a dropped answer leaves nothing behind — no failed request in the browser's network log, no error, no partial-result marker — and a reader could not tell a slow store from a lost response from a store with no streams. Every explorer read now has a 45-second deadline and reports a retryable failure naming what to narrow the read to, and the Refresh button no longer disables itself while a request is in flight.

The rollout panel could not always say whether a rollout was progressing, and said it anyway. Where every shard of a projection is unmeasurable, each version rendered a full row of blanks; the panel now collapses those to one line per projection, or to a single line for the whole panel when nothing on the service can be measured, and stamps the header with the snapshot's age when that snapshot is stale.

A projection command refused by a monitored service emitted no acknowledgement at all, so the console showed the action pending forever with the reason visible only in that service's log. Refusals now complete as refusals. Separately, Restart on a daemon-paused shard was a no-op that reported success — it lifted an operator pause and nothing else, so a shard the daemon had stopped stayed stopped while the console said the command had worked.

A pause no longer hides from the attention list. Paused shards were excluded from "needing attention", so the shard behind one August incident counted as healthy; a pause now needs attention unless an operator is the one who asked for it.

Ingest could latch under load. Per-service partitioning was not engaging on the highest-volume lane, so eighteen parallel upserts deadlocked and the console's shard view fell minutes behind. A retention trim deadlocked for a related reason — its ordering was not a total one, and every service trimmed the same window concurrently.

Ancillary event stores errored on every progression read and write, because they kept a bare progression table without the columns the console reads. CritterWatch now provisions that shape, and switches extended progression tracking on for every registered store through the store's own supported setting rather than inferring it from the container — see Upgrade Notes. A store that still reports the setting as off is now logged by name, which is a different failure from a read that returned nothing.

On a database-per-tenant PostgreSQL store the projection registry is readable now (marten#5382), so orphan detection works on the store shape that needs it most. The console reads that answer as a registry — every registered shard, no position — never as progression, so it cannot fork a shard onto an unattributed row.

Five field-reported defects carried over from the unreleased 1.0.2, all affecting fleets using something other than the default broker wiring:

  • The console failed to boot on a plain broker listener. A console wired the way the docs and samples show it died at startup on Wolverine 6.24+ with InvalidListenerConfigurationException: PartitionProcessingByGroupId() was configured on an Inline endpoint, naming a call the operator never made. CritterWatch applies that partitioning to every listener — it is the per-service single-writer guarantee — and Inline is simply the broker default. It now sets a mode that carries the partitioning, and leaves any listener you configured yourself alone. No configuration change is needed; if you added ProcessInParallelWithNativeAcks() as a workaround it remains correct and can stay.
  • Database-queue fleets silently stranded all telemetry. The console forced its Wolverine transport schema, so a transportSchema passed by an operator was overridden at host build. Services wrote telemetry, the console drained a different same-named table, the services endpoint returned [], and neither side logged anything. Your transportSchema wins again — see Upgrade Notes.
  • Telemetry over the HTTP transport was rejected with 415. The telemetry route had been switched to a buffered mode, which makes an HTTP endpoint post a batch, and only the batch endpoint accepts those. Corrected for 1.1.0: this said the documented /_wolverine/invoke target was correct again. It was not: that route cannot read CritterWatch's wire format, and nothing tested the channel end to end. 1.1.0 moves the HTTP channel to /_wolverine/batch/{queue}; see the 1.1.0 entry above.
  • The handshake log line named the wrong destination — a service reported sending its initial telemetry to its own control queue rather than to the telemetry URI, as did the publish-failure line, which is the one you read while actually debugging.
  • An all-SQL-Server or all-PostgreSQL database-queue console served 500 on every request (wolverine#4130). It booted and listened cleanly, then failed every request with … was NullMessageStore, because the outboxed session factory captured the Main message store before store reconciliation had assigned it. Fixed upstream and picked up by this release's pin. Store roles were correct throughout, which is why this presented as a packaging fault rather than a startup-ordering one; if you worked around it by registering the transport as ancillary, that remains a good configuration.

Supported stack ​

.NET9.0 and 10.0
Wolverine6.56.0
JasperFx2.96.0
Marten (PostgreSQL)9.54.1
Polecat (SQL Server)5.46.1
Fisher (SQLite)1.24.1
Weasel9.47.0
Vue3.5

Upgrading ​

A replay or discard that fails is now reported to the console instead of being thrown. The monitored service used to throw a failed DiscardAsync / ReplayAsync into its own error handling, where it was retried or dead-lettered under the dlq-operation message type, and the console never heard. Each database now answers with a DlqOperationCompleted carrying the error, plus the existing DeadLetterQueuePartialFailure, and the operation is not retried on the service. Run it again from the console.

Edit & Replay now needs dlq.edit as well as dlq.replay. It rewrites a dead letter's body before replaying it, and dlq.edit was declared for exactly that and checked nowhere, so anyone who could replay could also rewrite. On a host that registers an ICritterWatchAuthorizer, an operator who could edit-and-replay with dlq.replay alone is now refused, with a rejection naming dlq.edit. Grant it to whoever should keep the ability. Hosts without an authorizer are unaffected.

Read Upgrade Notes before moving a running console to 1.1 — this release changes the progression-table shape on monitored stores, and finishes the transport-schema change begun in 1.0.2.

Polecat.Mcp: MapPolecatMcp is now MapPolecatMcpServer. The old name collided with Polecat.AspNetCore's own, different MCP endpoint, MapPolecatMcp(this IEndpointRouteBuilder, ...), on the same default route, and overload resolution chose between them silently: with both namespaces imported you got this server, and with only Polecat.AspNetCore's you got the other. Rename the call from app.MapPolecatMcp() to app.MapPolecatMcpServer().

MapPolecatMcp still works in 1.1 as an [Obsolete] forwarder and is removed in 1.2. The route is unchanged (/polecat/mcp), so MCP clients need no change. MapMartenMcp and MapWolverineMcp had no collision and keep their names.

1.0.1 ​

Released 20 August 2026 — nine fixes from the first production deployments of 1.0.0, most of them cases where the console reported a confident number it had no basis for.

Fixed ​

No alert had ever resolved. 33 Critical alerts sat stuck for 15–17 days at version 0 with resolvedAt null, while the monitored application was provably caught up — the console could not append to its own streams, so the resolution half of the alert lifecycle never ran. The single most serious defect in 1.0.0: alerting that can raise but never clear is worse than no alerting, because the board stops being read.

Counts of the same thing disagreed with each other. The alert bell said 28, the Health tab badge said 29, and the API returned 33 — all active, all Critical, none acknowledged. Separately, the Operations tab rendered "Dead Letters 0" in green beside a sidebar badge reading 999+ — two different counts under one label on one screen.

"Max Lag 0 · all projections caught up" was computed from a gap the console could not compute. An uncomputable gap is not a zero one, and reporting it as zero turns a blind spot into a reassurance.

An active alert carried no evidence it was still true — 33 Criticals rendered identically whether the claim was 40 seconds or 17 days old. Alerts now say how old their claim is.

The self-monitoring failure counter could not see the failures it exists for.selfMonitoring.recentFailureCount read 0 through 214 hard message failures.

500 blank Type cells, silently. CwTypeCell was used in six files without being imported; Vue emits an unknown <cwtypecell> element rather than failing, so /raw and the Metrics tab rendered blank columns with nothing in the console to say why.

Page navigation blocked for 8–18 seconds at fleet scale. projectionsForService cost ~1.7 s per call, was not memoized, and was called 3–4× per navigation; /projections blocked 17.8 s, /events 9.8 s, /documents 8.2 s. Memoized per service, the same treatment that fixed the connect path in 1.0.0.

Eleven Element Plus deprecation warnings across the route sweep (el-pagination small, el-radio label), which break outright at Element Plus 3.0.

Upgrading ​

No configuration changes are required from 1.0.0.

1.0.0 ​

Released 19 August 2026 — the first stable release of CritterWatch.

CritterWatch is a monitoring and production-management console for Critter Stack systems. It gives you — and your AI agents — a live view of every Wolverine service, Marten and Polecat event store, and the controls to act on what you find.

1.0 marks the point where the API surface, the package lineup, and the storage model are stable and supported. Everything below has been through the full release gate on all three storage flavours.

Install ​

Pick the package matching the database you want CritterWatch's own console storage to use. This is independent of what your monitored services run — a console on SQLite can happily monitor a fleet of PostgreSQL-backed services.

PackageConsole storageWhen to reach for it
CritterWatchMarten / PostgreSQLThe default. Most Critter Stack shops already run Postgres
CritterWatch.SqlServerPolecat / SQL ServerShops standardised on SQL Server
CritterWatch.SqliteFisher / SQLiteLocal development, demos, and single-node deployments — needs no database server at all
bash
dotnet add package CritterWatch

Then see the Quick Start. Monitored services add Wolverine.CritterWatch, which is store-agnostic — it depends only on the JasperFx.Events abstractions, so a service on RavenDB, EF Core, or no event store at all can be monitored without pulling Marten in.

What is in 1.0 ​

Understand your system, not just watch it. CritterWatch is as much a development tool as a production one. It renders live event models, your real store schema and DDL, the generated handler and projection source Wolverine actually runs, and full HTTP endpoint chains — the ground truth rather than your mental model of it.

The Projection Stepper replays a projection over a real slice of your event store and walks you through the before/after state row by row, with a JSON diff at each step. It is the fastest available answer to "why is this aggregate shaped like that at version N?"

Real-time monitoring of service health, nodes, agents, throughput and error rates, pushed over SignalR as events happen rather than polled.

Dead letter queue management across every service from one view — query, filter, replay, edit-and-replay, discard, in batches, with exception grouping and CSV export.

Projection operations — track async projection lag, spot stalls, and pause, restart, rebuild or rewind from the console with no code change and no service restart.

Event-sourced alerting with a complete lifecycle — raised, elevated, reduced, resolved, cleared — stored as immutable events, so the alert history is auditable rather than a current-state table that forgets.

Multi-tenancy — add, disable and remove tenants at runtime, with per-tenant metrics, DLQ filtering and projection health.

An MCP server for AI agents. Point Claude or any MCP client at one endpoint and it can query every monitored service — spans by saga or stream, projection lag, dead letters, alerts, metrics — and take action, tenant-scoped and RBAC-gated.

Licensing ​

CritterWatch is free to use in read-only monitoring mode. Connect as many services as you like and watch your whole Critter Stack with no license key.

A commercial license unlocks every administrative action — DLQ replay, projection rebuilds, listener and node control, tenant management — and the entire MCP server.

Supported stack ​

1.0.0 is built and gated against:

.NET9.0 and 10.0
Wolverine6.29.1
Marten9.28.0
Polecat5.19.0
Fisher1.0.0
JasperFx / JasperFx.Events2.52.1

Upgrading from a release candidate ​

No configuration changes are required from 1.0.0-rc.10.

⚠️ If you are coming from a build older than rc.10 and ran an embedded console, note that CritterWatch.Embedded was removed before 1.0 — the console now runs as its own host. Its storage collapsed to a single store at the same time: console data lives in the critterwatch schema and Wolverine durability in critterwatch_wolverine. No table moved, but an upgraded database may still carry orphaned critterwatch.wolverine_* tables from the old two-store layout. Nothing reads them and they can be dropped.

Getting help ​

1.0 closed 282 issues across ten release candidates, beginning 27 July 2026.

Questions, bug reports and feature requests are welcome in the Critter Stack Discord, and licensed customers can reach us through the support address on their license.