Inbound Commands
Inbound commands are messages sent from the CritterWatch server to monitored services. They implement either IServiceMessage (fire-and-forget control commands) or IServiceQuery (request/response queries that return data) and are routed directly to the target service's CritterWatch listener.
All commands are automatically handled when AddCritterWatchMonitoring() is called — no additional handler registration is needed.
The signatures shown below are the live record declarations from Wolverine.CritterWatch/Messages/Inbound/*.cs. The companion await bus.SendAsync(...) line for each is the minimal call you'd issue from a custom integration; in CritterWatch's own UI flows these are emitted from the SignalR hub on operator action.
Operations gating: the console relays a mutating command only when its license enables operations (the Free tier is read-only), and, when the host registers an
ICritterWatchAuthorizer, only when the caller holds the command's capability. A refused command is answered withCommandRejectedand never reaches the service; an accepted one is audited and answered withCommandAccepted. Queries (IServiceQuery) are always allowed. The service checks its own license again before it acts — see License.
Every command carries a fixed [MessageIdentity], and the set of identities and members is a committed 1.x wire contract: a console and a service on different 1.x versions read each other's messages. A member a service does not know is ignored, so a newer console's option sent to an older service is dropped rather than refused.
This page shows the record declaration and a minimal call for the commands an integration is most likely to send itself. The rest of the set is listed under Other commands and queries.
DLQ Commands
DlqOperation
This is the command the console sends for every dead-letter action on the Dead Letters page and from the MCP dead-letter tools: DlqOperation(Type, ServiceName, Query, Databases), with Type one of Query, Summarize, Discard, Replay, EditAndReplay, SummarizeGroups, Archive or Restore, a Wolverine DeadLetterEnvelopeQuery (time range, message type, exception type and message, explicit message ids, page) and the message databases to run against (empty for all of them). Its message identity is dlq-operation. Optional members:
EditedMessageJson— the replacement body forEditAndReplay.RequestId— echoed on every answer, so the caller can tell its answers from another operator's.Reason— why the operator ran it. Recorded on the audit row and in the console's operation history. A 1.0 console never sends it.Grouping— howSummarizeGroupsgroups the dead letters:FailureGroup(the default when null),MessageType,EndpointorExceptionType.TargetServices— for an operation that names no service, only these services instead of every monitored one. Read by the console's fan-out; a service never acts on it.
SummarizeGroups, Archive and Restore are new in 1.1, and are numbered after the original five so older peers read the same values. The console sends SummarizeGroups only to a service whose client advertises dead-letter groups, and Archive / Restore only to one that advertises group operations; a 1.0 client is never sent them.
The service announces the databases it will run against with one DeadLetterQueueFanOut, then runs the operation against each one. A Query or Summarize answers with DeadLetterDetails or DeadLetterQueueSummaryResults per database, and a SummarizeGroups with DeadLetterGroupSummaryResults. A Replay, Discard, EditAndReplay, Archive or Restore answers with a DlqOperationCompleted per database, carrying how many dead letters it affected (null when the store does not report a count) or the error. A database that fails also answers with a DeadLetterQueuePartialFailure, and the operation is not retried on the service — run it again from the console. The console follows each mutating operation as a job on the Operations tab and at GET /api/critterwatch/dlq/operations.
Mutating operations need the dlq.replay, dlq.discard or dlq.archive (for Archive and Restore) capability when permissions are wired; EditAndReplay needs dlq.edit as well as dlq.replay.
CancelDlqOperation
CancelDlqOperation(ServiceName, RequestId), message identity cancel-dlq-operation, stops a running replay, discard, archive or restore: the databases that have not started yet answer DlqOperationCompleted with Cancelled instead of running, while a database already running its statement finishes. The node that receives it forwards it to the service's other nodes. The console authorizes a cancel against the capability of the operation it stops, and relays it only to the services still working on that operation whose client advertises group operations.
ReplayMessages
Replay specific dead-lettered messages back through the processing pipeline. The IDs are the envelope ids returned from a DLQ summary or query.
The console itself never sends ReplayMessages or DiscardMessages — it sends DlqOperation. They remain this package's programmatic API for an application that wants to replay or discard by id itself.
[MessageIdentity("replay_messages")]
public record ReplayMessages(string ServiceName, Uri StoreUri, Guid[] IdList) : IServiceMessage;await bus.SendAsync(new ReplayMessages(
"trip-service",
new Uri("postgresql://localhost/trip-db"),
[envelopeId]));DiscardMessages
Permanently remove dead-lettered messages from the queue.
[MessageIdentity("discard_messages")]
public record DiscardMessages(string ServiceName, Uri StoreUri, Guid[] IdList) : IServiceMessage;await bus.SendAsync(new DiscardMessages(
"trip-service",
new Uri("postgresql://localhost/trip-db"),
[envelopeId]));EditScheduledMessage
Edit a scheduled message's body and/or scheduled time before it fires. Either edit may be null. The MessageId is the envelope id of the scheduled message.
[MessageIdentity("edit_scheduled_message")]
public record EditScheduledMessage(
Guid MessageId,
string? EditedBodyJson,
DateTimeOffset? NewScheduledTime) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new EditScheduledMessage(
messageId,
EditedBodyJson: """{"amount":42}""",
NewScheduledTime: DateTimeOffset.UtcNow.AddHours(1))
{ ServiceName = "trip-service" });Listener Commands
PauseListener
Pause consumption on a single receiving endpoint across the entire application, durably. Every node stops pulling messages off that queue, and the pause is persisted as an agent restriction that survives restarts and new nodes joining the cluster — it stays paused until you RestartListener, with no auto-resume. Messages stay safely on the broker. In-flight work is not waited on; for a graceful shutdown that lets queued and in-flight messages finish first, use DrainListener instead.
What the pause reaches
An endpoint is not a Wolverine agent, so the persisted pause only stops a node from starting the listener. The node that handles the command therefore asks every other running node to pause it too, over each node's control endpoint. If a node does not apply it, for instance one still on an older CritterWatch during a rolling deploy, the command fails with CommandFailed and names that node. The pause stands on every other node, and the named node goes on consuming until it restarts.
Local queues differ. A durable local queue pauses on every running node, but it does not read the persisted pause, so it comes back accepting on any host that starts while the pause stands. A buffered local queue has no way to stop, so the command is refused with CommandFailed and nothing changes. An exclusive listener pauses on the one node that runs it.
A URI that no listener of the service answers to, a mistyped queue name for instance, is refused the same way, and nothing is persisted.
(Under single-node / DurabilityMode.Solo hosts, where there is no cluster to coordinate, this falls back to latching the local listener — the effect is the same: consumption stops until resumed.)
[MessageIdentity("pause_listener")]
public record PauseListener(string ServiceName, string EndpointUri) : IServiceMessage;await bus.SendAsync(new PauseListener("trip-service", "rabbitmq://exchange/trips/"));RestartListener
Resume a durably-paused (or drained) listener: clears the pause restriction across the application and restarts consumption on every node, including a node that started while the pause stood. The accumulated backlog on the broker drains down once consumption resumes. An exclusive listener restarts only on the node that runs it.
If a node does not apply the restart, the command fails with CommandFailed and names that node, which resumes when it restarts.
A URI that no listener of the service answers to still has any persisted pause for it lifted, which is how a pause left behind by an endpoint a deploy removed is cleared. The command then fails with CommandFailed, saying there was nothing to restart.
[MessageIdentity("restart_listener")]
public record RestartListener(string ServiceName, string EndpointUri) : IServiceMessage;await bus.SendAsync(new RestartListener("trip-service", "rabbitmq://exchange/trips/"));DrainListener
Graceful shutdown of a single listener — stops accepting new messages, lets queued and in-flight messages finish, then halts. The right call before a deploy or a planned-maintenance window. Distinct from PauseListener which abandons in-flight work. A URI that no listener of the service answers to is refused with CommandFailed.
[MessageIdentity("drain_listener")]
public record DrainListener(string ServiceName, string EndpointUri) : IServiceMessage;await bus.SendAsync(new DrainListener("trip-service", "rabbitmq://exchange/trips/"));PauseAllListeners
Pause every transport listener on the node that handles the command, except the control channel it arrived on. Useful before an emergency maintenance window.
This is not PauseListener applied to every endpoint. Nothing is persisted, so the listeners resume when the host restarts, other nodes keep consuming, and local queues are not paused at all. To pause one endpoint on every node, durably, use PauseListener.
[MessageIdentity("pause_all_listeners")]
public record PauseAllListeners(string ServiceName) : IServiceMessage;await bus.SendAsync(new PauseAllListeners("trip-service"));RestartAllListeners
Resume every listener in the service, on every node. It clears every persisted listener pause, including a pause made with PauseListener, and restarts consumption across the application. Paused projections and subscriptions are not listeners and stay paused.
[MessageIdentity("restart_all_listeners")]
public record RestartAllListeners(string ServiceName) : IServiceMessage;await bus.SendAsync(new RestartAllListeners("trip-service"));Endpoint Configuration Commands
UpdateEndpointBufferingLimits
Adjust the in-memory buffering thresholds on a single endpoint at runtime. Maximum is MaximumMessagesToReceive; Restart is the recovery threshold below which buffering resumes.
[MessageIdentity("update_endpoint_buffering_limits")]
public record UpdateEndpointBufferingLimits(string EndpointUri, int Maximum, int Restart) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new UpdateEndpointBufferingLimits(
"rabbitmq://exchange/trips/",
Maximum: 1000,
Restart: 200)
{ ServiceName = "trip-service" });UpdateEndpointCircuitBreaker
Adjust the circuit-breaker configuration on a single endpoint at runtime.
[MessageIdentity("update_endpoint_circuit_breaker")]
public record UpdateEndpointCircuitBreaker(string EndpointUri, double FailurePercentageThreshold, TimeSpan PauseTime) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new UpdateEndpointCircuitBreaker(
"rabbitmq://exchange/trips/",
FailurePercentageThreshold: 0.25,
PauseTime: TimeSpan.FromMinutes(2))
{ ServiceName = "trip-service" });Projection Commands
All four projection commands accept an optional TenantId. Leave it null (the default) to act on the store-global shard — the existing single-tenant behavior, byte-for-byte. Set it to a tenant id to scope the operation to that one tenant's shard for the projection. The CritterWatch UI fills it in automatically based on the Tenants-tab row you act from; integrations sending these commands directly (custom UIs, ops scripts, the HTTP API) need to set it themselves.
Per-tenant operations respect tenant-scoped RBAC — a user authorized for only some tenants can't rebuild another tenant's projection. See Multi-Tenancy → Per-tenant scoping for the bigger picture.
PauseProjection
Pause a single projection shard. The shard stops advancing until restarted.
[MessageIdentity("pause_projection")]
public record PauseProjection(string ServiceName, Uri AgentUri) : IServiceMessage
{
public string? TenantId { get; init; }
/// <summary>Client-supplied correlation id, echoed back on the ProjectionCommandAck so the
/// UI can map the acknowledgement to the originating action. Null for non-UI callers.</summary>
public string? RequestId { get; init; }
}await bus.SendAsync(new PauseProjection(
"trip-service",
new Uri("marten://projection/TripSummary:All")));
// Per-tenant variant
await bus.SendAsync(new PauseProjection(
"trip-service",
new Uri("marten://projection/TripSummary:All"))
{ TenantId = "acme-corp" });RestartProjection
Resume a paused projection shard.
[MessageIdentity("restart_projection")]
public record RestartProjection(string ServiceName, Uri AgentUri) : IServiceMessage
{
public string? TenantId { get; init; }
/// <summary>Client-supplied correlation id, echoed back on the ProjectionCommandAck so the
/// UI can map the acknowledgement to the originating action. Null for non-UI callers.</summary>
public string? RequestId { get; init; }
}await bus.SendAsync(new RestartProjection(
"trip-service",
new Uri("marten://projection/TripSummary:All")));RebuildProjection
Reset a projection's state and rebuild from the beginning of its source stream. Long-running on big stores; while it runs, the shard's threshold-based alerts should be suppressed via Alert Configuration → Projections → Per-Shard Overrides.
TenantId rebuilds only that tenant's read model — useful when a single tenant's projection has drifted (a code bug, a manual data fix) and you don't want to spend the rebuild cost on every tenant. To rebuild every tenant's partition, the console's Rebuild all tenants row action fans out one per-tenant rebuild per tenant — see Multi-Tenancy.
[MessageIdentity("rebuild_projection")]
public record RebuildProjection(string ServiceName, Uri? AgentUri) : IServiceMessage
{
/// <summary>
/// Name-addressed rebuild target for a projection that has no live agent URI (an <c>Inline</c> or
/// otherwise non-distributed projection). Set this (with <see cref="AgentUri"/> null) to resolve the
/// projection by name via the transient rebuild path. The handler composes the shard identity with
/// <c>ShardName.Compose(ProjectionName)</c> — never string concatenation.
/// </summary>
public string? ProjectionName { get; init; }
/// <summary>
/// The event store the named projection lives in. Optional; carried for store-scoped resolution of
/// a name-addressed rebuild. (The transient rebuild path is not yet store-scoped — a projection name that
/// exists in two stores rebuilds the first match; narrowing needs an upstream store-scoped overload.)
/// </summary>
public Uri? StoreUri { get; init; }
/// <summary>
/// The registered projection <b>version</b> of a name-addressed (<see cref="ProjectionName"/>)
/// rebuild target. A projection registered at V2+ carries a versioned shard identity
/// (<c>Name:V2:All</c>), and the transient rebuild path matches the target on that full identity — so a
/// V1-composed <c>Name:All</c> would never resolve a V2 projection. When set (and greater than 1) the
/// handler composes the shard identity with <c>ShardName.Compose(ProjectionName, version: Version)</c>
/// so the correct registered version is matched. Null/1 preserves the single-version common case
/// verbatim. Only meaningful for a name-addressed rebuild — an agent-addressed rebuild already carries
/// the version through its live agent URI.
/// </summary>
public int? Version { get; init; }
/// <summary>
/// When non-null, the rebuild is scoped to
/// the named tenant via <c>ShardName.ForTenant(name, tenantId)</c>
/// and dispatched to the per-tenant daemon API
/// (<c>RebuildProjectionAsync(name, tenantId, …)</c>). Null
/// preserves the existing store-global rebuild verbatim.
/// </summary>
public string? TenantId { get; init; }
/// <summary>Client-supplied correlation id, echoed back on the ProjectionCommandAck so the
/// UI can map the acknowledgement to the originating action. Null for non-UI callers.</summary>
public string? RequestId { get; init; }
}// Single-tenant service — or a global rebuild on a multi-tenant store.
await bus.SendAsync(new RebuildProjection(
"trip-service",
new Uri("marten://projection/TripSummary:All")));
// Per-tenant rebuild
await bus.SendAsync(new RebuildProjection(
"trip-service",
new Uri("marten://projection/TripSummary:All"))
{ TenantId = "acme-corp" });RewindSubscription
Move a subscription's position back to a chosen point. The mode controls which target field applies.
[MessageIdentity("rewind_subscription")]
public record RewindSubscription(
string AgentUri,
RewindMode Mode,
long? TargetSequence,
DateTimeOffset? TargetTimestamp) : IServiceMessage
{
public string ServiceName { get; init; } = "";
/// <summary>
/// When non-null, the rewind is scoped to
/// the named tenant. Null preserves the existing store-global
/// rewind verbatim.
/// </summary>
public string? TenantId { get; init; }
/// <summary>Client-supplied correlation id, echoed back on the ProjectionCommandAck so the
/// UI can map the acknowledgement to the originating action. Null for non-UI callers.</summary>
public string? RequestId { get; init; }
/// <summary>
/// Set only on the per-tenant rewinds a tenant-less rewind fans out into on a store that partitions
/// events per tenant: the RequestId of the rewind they came from, echoed on every ack so the console can count
/// k of N. Never set by the console.
/// </summary>
public string? ParentRequestId { get; init; }
}
public enum RewindMode
{
ToBeginning,
ToSequence,
ToTimestamp
}await bus.SendAsync(new RewindSubscription(
"marten://subscription/AnalyticsExporter",
RewindMode.ToTimestamp,
TargetSequence: null,
TargetTimestamp: DateTimeOffset.UtcNow.AddHours(-1))
{ ServiceName = "trip-service" });On a store that partitions events per tenant (Marten UseTenantPartitionedEvents, which reports IEventStore.DistributesAgentsPerTenant and carries the per-tenant-agents capability tag), every tenant has its own agent and its own event sequence:
- With a
TenantId, only that tenant's agent is rewound, and aToTimestampfloor is resolved against that tenant's own events. This needs JasperFx 2.91 and Marten 9.51 on the monitored service. An older stack refuses it, and the command fails with a message saying so rather than the daemon'sNotSupportedException. - Without a
TenantId, the service fans the command out into one per-tenantRewindSubscriptionfor every tenant agent of the subscription that the cluster is running, read from the node table. Each one is forwarded to the node that owns that tenant's agent. The command's ownProjectionCommandAckcompletes withFanOutCountset to the number of tenants, and is not the outcome. Each tenant then acks for itself withParentRequestIdset to the command'sRequestId, so the console shows k of N until every tenant has reported. Each tenant also gets its own timeline entry. ToSequencewithout aTenantIdis refused, both here and byScheduleSubscriptionRewindwhen the rewind is scheduled rather than when it would run. A sequence number is only a position within one tenant's events.
The fan-out never falls through to the daemon's own tenant-less rewind. That rewind reaches only the tenants its node runs, and on a node that runs none of them it starts a store-global agent that reads every tenant's sequences. If no node is running a tenant agent for the subscription, the command fails instead.
EjectProjection
Permanently delete the leftover progression row for an orphaned shard — one whose projection has been renamed, versioned, or removed, so it shows up as an "Orphaned" card and can never be paused, restarted, or rebuilt (no registered projection owns it). Ejecting drops the row outright through the store-agnostic IEventDatabase.DeleteProjectionProgressByShardNameAsync (Marten 9.11 / Polecat 4.6); the orphan card then reconciles out of the UI on the next progression poll. The match is on the exact shard identity, so siblings are never collaterally dropped, and a non-existent identity is a clean no-op. This is irreversible — a still-registered projection would simply re-create its row from zero, so only eject genuine orphans.
StoreUri targets the owning store on a multi-store (ancillary) service; omit it on single-store services.
[MessageIdentity("eject_projection")]
public record EjectProjection(string ServiceName, string ShardName) : IServiceMessage
{
/// <summary>
/// The owning store's clean identity (the <c>marten://…</c> / <c>polecat://…</c> <c>StoreUri</c> the
/// progression poller stamps on each shard state). Selects the registered store whose
/// <c>IEventStore.Subject</c> matches — needed on multi-store (ancillary) services to avoid ejecting a
/// same-named row from the wrong store. Null on single-store services ⇒ every registered store is
/// targeted (the delete is an exact-identity, zero-row no-op against stores that don't hold the row).
/// </summary>
public string? StoreUri { get; init; }
}// Delete the leftover progression row for an orphaned shard (a renamed / versioned /
// removed projection). Pass the raw shard identity exactly as surfaced on the Orphaned card.
await bus.SendAsync(new EjectProjection(
"trip-service",
"TripSummary:V1:All"));
// On a multi-store (ancillary) service, name the owning store so a same-named row in
// another store is left untouched.
await bus.SendAsync(new EjectProjection(
"trip-service",
"TripSummary:V1:All")
{ StoreUri = "marten://main" });EjectRetiredProjectionVersion
Permanently delete every progression row of one retired projection version, on every database and tenant of the targeted store(s). A version bump leaves the old version's rows behind, and the stores never delete them; on a tenant-partitioned projection that is one row per (database, tenant), thousands per deploy. Where EjectProjection names one row, this names a version, and the service enumerates the rows itself from each database's own progression table.
Two gates keep it from touching a projection that is still running:
- The console relays it only for a version it has collapsed: one whose every shard has been orphaned for 24 hours, which the shard-progression summary lists with a
collapsedAt. During a blue/green rollout each fleet reports the other fleet's live shards as orphans, so an orphaned shard alone is not proof that nothing runs it. - The service refuses the whole command, deleting nothing, if it still registers that projection version or cannot read its projection registry.
The service answers with RetiredProjectionVersionEjected: rows deleted, databases touched, or the reason it refused. On success the console drops its collapsed record. A service whose Wolverine.CritterWatch package predates 1.1 has no handler for this command and never answers. Irreversible.
[MessageIdentity("eject_retired_projection_version")]
public record EjectRetiredProjectionVersion(string ServiceName, string ProjectionName, int Version) : IServiceMessage
{
/// <summary>
/// The owning store's identity (the <c>marten://…</c> / <c>polecat://…</c> <c>StoreUri</c>), to target one store
/// on a multi-store service. Null targets every registered store; a store without the version deletes nothing.
/// </summary>
public string? StoreUri { get; init; }
}// Delete every progression row a retired projection version left behind, on every
// database and tenant: name the projection WITHOUT its version, and the version.
// The console relays this only for a version it has collapsed (every shard orphaned
// for 24 hours), and the service refuses a version it still registers.
await bus.SendAsync(new EjectRetiredProjectionVersion(
"trip-service",
"TripSummary",
7));Agent Commands
PinAgentToNode
Force an agent to run on a specific node and prevent the leader from rebalancing it elsewhere. Useful when a particular node has hardware or licensing affinity (a GPU, a license-restricted IP, etc.).
[MessageIdentity("pin_agent_to_node")]
public record PinAgentToNode(string ServiceName, Uri AgentUri, int NodeNumber) : IServiceMessage;await bus.SendAsync(new PinAgentToNode(
"trip-service",
new Uri("marten://projection/TripSummary:All"),
NodeNumber: 2));UnpinAgent
Remove a pin so the leader can rebalance the agent again.
[MessageIdentity("unpin_agent")]
public record UnpinAgent(string ServiceName, Uri AgentUri) : IServiceMessage;await bus.SendAsync(new UnpinAgent(
"trip-service",
new Uri("marten://projection/TripSummary:All")));PushAgentThresholds
Push updated behind-high-water-mark warning + critical thresholds to a specific shard for client-side validation. Mirrors the values stored in Alert Configuration → Projections → Per-Shard Overrides so the service can refuse work that would never satisfy the operator's stated SLO.
[MessageIdentity("push_agent_thresholds")]
public record PushAgentThresholds(string ShardName, long? WarningBehind, long? CriticalBehind) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new PushAgentThresholds(
"TripSummary:All",
WarningBehind: 1000,
CriticalBehind: 10000)
{ ServiceName = "trip-service" });RequestAgentHealthReport (query)
Ask the service to immediately publish an agent-health snapshot, bypassing the periodic health-report timer. Implements IServiceQuery.
[MessageIdentity("request_agent_health_report")]
public record RequestAgentHealthReport : IServiceQuery
{
public string ServiceName { get; init; } = "";
}await bus.InvokeAsync(new RequestAgentHealthReport { ServiceName = "trip-service" });RequestHandlerSourceCode (query)
Ask the service to send the generated handler source code for a given message type. The optional EndpointUri selects the endpoint-specific specialisation when sticky routing applies.
[MessageIdentity("request_handler_source_code")]
public record RequestHandlerSourceCode(string HandlerMessageType, string? EndpointUri) : IServiceQuery
{
public string ServiceName { get; init; } = "";
}await bus.InvokeAsync(new RequestHandlerSourceCode(
"Trips.CompleteTrip",
EndpointUri: null)
{ ServiceName = "trip-service" });RequestEndpointProperties (query)
Lazy-fetch the Properties + Children configuration tree for one Wolverine endpoint. The Pipeline tab issues this when the operator opens an endpoint row; the response is cached on the console keyed by service version. See Observer → Lazy-fetched detail panes for the full caching shape.
[MessageIdentity("request_endpoint_properties")]
public record RequestEndpointProperties(string EndpointUri) : IServiceQuery
{
public string ServiceName { get; init; } = "";
}await bus.InvokeAsync(new RequestEndpointProperties(
"rabbitmq://exchange/trips/")
{ ServiceName = "trip-service" });RequestMessageHandlerProperties (query)
Lazy-fetch the per-handler Properties rows for one message type. The handler-chain detail page issues this when opened. Returns one row per handler chain registered against the message type.
[MessageIdentity("request_message_handler_properties")]
public record RequestMessageHandlerProperties(string MessageType) : IServiceQuery
{
public string ServiceName { get; init; } = "";
}await bus.InvokeAsync(new RequestMessageHandlerProperties(
"Trips.CompleteTrip")
{ ServiceName = "trip-service" });Node Commands
EjectNode
Remove one or more stale nodes from the cluster. The leader redistributes their assigned agents.
[MessageIdentity("eject_node")]
public record EjectNode(string ServiceName, int[] NodeNumbers) : IServiceMessage;await bus.SendAsync(new EjectNode("trip-service", [3]));TriggerElection
Force a new leader election. The current leader's lease is revoked and the cluster votes again. Disruptive — only fire when the cluster is genuinely stuck.
[MessageIdentity("trigger_election")]
public record TriggerElection(string ServiceName) : IServiceMessage;await bus.SendAsync(new TriggerElection("trip-service"));ClearNodeHistory
Trim the historical-node log, retaining the most recent RetainRecords entries. The optional Timestamp is the cutoff before which records are eligible for removal.
[MessageIdentity("clear_node_history")]
public record ClearNodeHistory(string ServiceName, int RetainRecords, DateTimeOffset? Timestamp) : IServiceMessage;await bus.SendAsync(new ClearNodeHistory(
"trip-service",
RetainRecords: 10,
Timestamp: null));Tenant Commands
All tenant commands carry an implicit ServiceName via IServiceMessage's ServiceName property; the explicit parameters are just the tenant fields.
AddTenant
Add a new tenant database to a multi-tenant service. The connection string is sent over SignalR; rotate credentials after the cutover if you want a clean audit trail.
[MessageIdentity("add_tenant")]
public record AddTenant(string TenantId, string ConnectionString) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new AddTenant(
"acme-corp",
"Host=db;Database=acme;Username=...;Password=...")
{ ServiceName = "trip-service" });DisableTenant
Soft-disable a tenant. Messages addressed to the tenant queue without losing data; can be re-enabled with EnableTenant.
[MessageIdentity("disable_tenant")]
public record DisableTenant(string TenantId) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new DisableTenant("acme-corp") { ServiceName = "trip-service" });EnableTenant
Re-enable a previously disabled tenant.
[MessageIdentity("enable_tenant")]
public record EnableTenant(string TenantId) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new EnableTenant("acme-corp") { ServiceName = "trip-service" });RemoveTenant
Remove a tenant's master-table record. The per-tenant database itself is not dropped — it stays around for any forensic work or restore scenario.
[MessageIdentity("remove_tenant")]
public record RemoveTenant(string TenantId) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new RemoveTenant("acme-corp") { ServiceName = "trip-service" });HardDeleteTenant
Drop the tenant database and remove the master-table record. Permanent. The CritterWatch UI gates this behind a typed-tenant-id confirmation modal — see Services → Tenants tab. License-gated to Professional+ on multi-tenant deployments.
[MessageIdentity("hard_delete_tenant")]
public record HardDeleteTenant(string TenantId) : IServiceMessage
{
public string ServiceName { get; init; } = "";
}await bus.SendAsync(new HardDeleteTenant("acme-corp") { ServiceName = "trip-service" });RequestTenantList (query)
Ask the service to publish its current tenant list. Used by the Tenants tab's Refresh button. Implements IServiceQuery.
[MessageIdentity("request_tenant_list")]
public record RequestTenantList : IServiceQuery
{
public string ServiceName { get; init; } = "";
}await bus.InvokeAsync(new RequestTenantList { ServiceName = "trip-service" });Other commands and queries
Every command below is handled automatically by AddCritterWatchMonitoring(), like the ones above. The record declarations are in Wolverine.CritterWatch/Messages/Inbound/, DeadLetterQueues/, ScheduledMessages/ and EventStores/.
Projections and high water
| Command | What it does |
|---|---|
ScheduleProjectionRebuild(ServiceName, AgentUri, RunAt) | Rebuild a projection later. The schedule lives in the service's own message store, so it shows on the Scheduled Messages page and can be cancelled there. |
ScheduleSubscriptionRewind | Rewind a subscription later, to the beginning, a sequence or a timestamp. A sequence needs a tenant on a partitioned store. |
RestartHighWaterDetection | Restart one database's high water detection loop on one node. Never moves the mark. Answered by HighWaterDetectionCommandAck. |
PreviewHighWaterAdvance / AdvanceHighWaterMark | Show what advancing a database's mark past a gap would skip, then advance it — only if the mark is still at ExpectedCurrentMark and, when it is set, the gap still skips ExpectedSkipCount sequences. |
Recurring schedules and stream compaction
| Command | What it does |
|---|---|
PauseRecurringSchedule(ServiceName, ScheduleName) | Stop a schedule producing occurrences and cancel the one already scheduled. |
ResumeRecurringSchedule(ServiceName, ScheduleName) | Resume it. Missed occurrences are not backfilled. |
TriggerRecurringSchedule(ServiceName, ScheduleName) | Run it once now without changing its cadence. |
RunCompactionPolicy(ServiceName, PolicyName) | Run a compaction policy now, as declared: a dry-run policy dry-runs. |
All four answer with a RecurringScheduleCommandAck or CompactionPolicyCommandAck.
Scheduled messages
| Command | What it does |
|---|---|
ScheduledMessageOperation | Query or summarize scheduled messages, per message database. |
CancelScheduledMessages / RescheduleScheduledMessage | Cancel scheduled messages, or move one to a new time. |
Documents
| Command | What it does |
|---|---|
RequestDocumentTypes / RequestDocuments / RequestDocument | The Document Explorer's reads. Need EnableDocumentExplorer. |
RequestDocumentHealth | A bounded health check of one document type: documents its serializer refuses, and members that changed between the oldest and newest documents. Needs EnableDocumentExplorer. |
RequestDocumentTenantCounts | Counts one document type per tenant under the explorer's filter: at most 500 tenants, one count query each, under one time budget. A tenant that cannot be counted is reported with its error, never as zero. Needs EnableDocumentExplorer. |
PreviewDocumentSave | A dry run of a save: what the store would keep, or why it would refuse. |
SaveDocument / DeleteDocument / UndeleteDocument | Replace a document from its JSON, delete one (soft or permanent, as the store is configured), or restore a soft-deleted one. Need EnableDocumentEditing. |
RequestStoreDocumentMappingDdls | The DDL a document store generates for its types. |
Event store, sagas and diagnostics (queries)
| Command | What it fetches |
|---|---|
EventStoreOperation | Stream fetch, by-metadata queries and stream compaction for the Event Store Explorer. |
RequestRecentStreams / RequestStreamEvents / RequestStreamMetadata / RequestEventsByTags | Event Store Explorer reads. Need EnableEventStoreExplorer. |
RequestSqlQuery | One read-only SELECT, guarded and row-capped. Needs EnableSqlQuery. |
RequestRehydrateAggregate | An aggregate at a given version. |
RequestProjectionRun | Step a projection over a slice of events, for the Projection Stepper. |
RequestProjectionStatuses / RequestProjectionDeadLetters / RequestProjectionSourceCode | Projection status, projection dead letters, and generated projection source. |
RequestSagaInstance / RequestSagaInstances | Saga state. |
RequestEventModel | The service's Event Model. |
RequestRoutingExplanation | Why a message type routes where it does. |
RequestHttpChainSourceCode / RequestHttpChainOpenApi | Generated source and OpenAPI for an HTTP chain. |
RequestPendingMigrationsCheck | Pending EF Core migrations for a DbContext. |
RequestCapabilities | Resend the service's capabilities. |
Chaos monkey (sample and test services)
EnableChaosMonkey, DisableChaosMonkey, SetChaosMonkeyFailureRate, SetChaosMonkeySlowHandler, SetChaosMonkeyProjectionFailureRate, SetChaosMonkeyProjectionPoison, ClearChaosMonkeyProjectionPoison, GetChaosMonkeyStatus and SeedDeadLetters.
What each one publishes back is on Outbound Events.
See Also
- Outbound Events — events services publish to CritterWatch
- Registration — installing the integration in your service
- Multi-Tenancy — full tenant management flow
- Audit Log — every command emits an audit entry; use the log to reconstruct what was sent and when
