Stream Compaction Policies

Event streams grow forever. Compaction collapses a stream's history into a single Compacted<T> snapshot: the events are folded into the aggregate, the snapshot replaces them, and the rest are deleted. It is invisible from the read side — the stream still aggregates to the same state and keeps its version — so what you get back is a smaller database and a cheaper replay.

A stream compaction policy does that declaratively across a whole store:

"Compact any stream of aggregate type Trip whose un-compacted growth exceeds 1,000 events, every night at 02:00."

WARNING

Compaction is destructive and irreversible, and its effects are invisible afterwards. A stream compacted with a threshold that was an order of magnitude too low looks exactly like one that was not: it aggregates to the same state, its version is unchanged, and nothing downstream reports a problem. That is why a new policy is a dry run until you arm it.

Store support

Store	Selecting streams	Compacting them
Marten (PostgreSQL)	✅	✅
Fisher (SQLite)	✅	✅
Polecat (SQL Server)	✅	✅ — since Polecat 5.25.0

All three stores compact. Polecat could not until 5.25.0 — it implemented the typed compaction path but not the untyped one a policy needs, so it selected the right streams and then refused every one of them (polecat#572). On an older Polecat you will still see that.

A store that cannot compact is reported once per run as a store-capability error rather than as a failure per stream, so it reads as what it is — a gap in the store, not a problem with your data or your policy.

Declaring a policy

Policies are declared in code, on the monitored service, and chained off AddCritterWatchMonitoring:

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.Services.AddMarten(m =>
            {
                m.Connection("...");

                // REQUIRED on Marten: compaction folds a stream into a Compacted<T> snapshot,
                // so the store must know how to aggregate T. Without a registered projection
                // for the policy's aggregate type, every stream refuses.
                m.Projections.Snapshot<Trip>(SnapshotLifecycle.Inline);
            })
            .IntegrateWithWolverine();

        opts.AddCritterWatchMonitoring(
                new Uri("rabbitmq://queue/critterwatch"),
                new Uri("rabbitmq://queue/critterwatch-control"))
            .CompactStreams(p =>
            {
                p.Name = "NightlyTrips";
                p.AggregateType = typeof(Trip);

                // Compact once a stream has grown more than 1,000 events SINCE its last
                // compaction — not once it passes 1,000 events in total.
                p.GrowthThreshold = 1000;

                p.Cron = "0 2 * * *";

                // The default. Leave it until a dry run has told you what the policy would
                // actually do to this database, then arm it deliberately.
                p.DryRun = true;
            });
    }).StartAsync();

NOTE

A policy can name any aggregate type — it does not have to be one your service already snapshots. Compaction folds the stream into a Compacted<T> snapshot, and naming T in the policy is itself the declaration of intent, so no registered aggregation projection is required.

Marten used to require one, refusing every stream with "Unable to find an Aggregation Projection for type X" while Fisher compacted the same aggregate happily. That divergence was reported as jasperfx#800 and settled against the refusal: requiring a registration would have confined policies to already-snapshotted aggregates, which is a constraint invisible until runtime and a portability gap between stores rather than a safety property. Marten 9.33.0 and Fisher 1.3.0 both behave this way. On an older Marten the old refusal still applies.

Growth, not version

GrowthThreshold measures events since the last compaction — Version - CompactedVersion — not the stream's total length. This matters:

• A stream at version 5,000 that was fully compacted at 5,000 has grown 0. It is not selected.
• The same stream after two more appends has grown 2.

A policy keyed on raw Version would re-compact the same streams on every single tick, forever.

Dry run, then arm

A new policy has DryRun = true. On each tick it reports how many streams it would compact, with a sample of their identities, and changes nothing.

Read that report, satisfy yourself the threshold is right, then set DryRun = false and redeploy. You do not have to wait for 02:00 to see it — Run now evaluates the policy immediately, and the dry-run report lands on the timeline straight away.

using var host = await Host.CreateDefaultBuilder()
    .UseWolverine(opts =>
    {
        opts.AddCritterWatchMonitoring(
                new Uri("rabbitmq://queue/critterwatch"),
                new Uri("rabbitmq://queue/critterwatch-control"))
            .CompactStreams(p =>
            {
                p.Name = "NightlyTrips";
                p.AggregateType = typeof(Trip);
                p.GrowthThreshold = 1000;
                p.Cron = "0 2 * * *";

                // Armed: this policy will delete events.
                p.DryRun = false;

                // Evaluate once per registered tenant. Mutually exclusive with p.TenantId.
                p.AllTenants = true;

                // Bound one tick's blast radius. A backlog is worked through over several
                // ticks rather than in one sweep.
                p.MaxStreamsPerTick = 250;

                // Never compact these, whatever the selector says.
                p.ExcludedStreams = ["trip-legal-hold-4471"];
            });
    }).StartAsync();

Options

Option	Default	What it does
Name	—	Unique within the service. Names the schedule, the console row, and the MCP addressing.
AggregateType	—	The aggregate whose streams are selected.
GrowthThreshold	—	Compact once un-compacted growth exceeds this. Must be at least 1.
Cron	—	5-field, or 6-field with leading seconds. Validated where you write it.
TimeZone	Wolverine's default	Zone the cron is interpreted in.
DryRun	true	Report only; change nothing.
MaxStreamsPerTick	100	Bounds one tick's blast radius.
TenantId	none	Scope to one tenant. Mutually exclusive with AllTenants.
AllTenants	false	Evaluate once per registered tenant.
ExcludedStreams	empty	Never compact these, whatever the selector says.

NOTE

Archived streams are never compacted. The selector excludes them, and there is no option to include them.

About MaxStreamsPerTick

It is a safety rail, not a tuning knob. It bounds what a first armed run can do, and it keeps one tick from monopolising the store. A backlog is worked through over several ticks, which is intended.

It matters most on Fisher/SQLite, which takes one writer per file: an uncapped sweep there does not merely run long, it blocks every other writer in the process for the duration.

How it runs

Each policy is registered as its own Wolverine recurring schedule, named critterwatch-compaction:{PolicyName}. That is why a policy behaves like every other scheduled job in the fleet:

• It fires once per cluster, on the leader, with per-occurrence deduplication.
• It appears on the console's Schedule Explorer, alongside your application's own schedules.
• It can be paused and resumed from the console or over MCP, without a redeploy.

WARNING

Declaring your first policy turns Wolverine's recurring-message feature on, which provisions the wolverine_recurring_messages tracking table. A service that declares no policy gets no schedule, no agent, and no schema change — adding the CritterWatch package alone never touches your database in this way.

What an operator can change without a redeploy

NOTE

The console's Run-now button is the one piece still to land (#1199). Everything else works, including all five MCP tools.

Policies are code-first, so the definition — threshold, cron, aggregate type, and the DryRun flag — changes only by editing the declaration and redeploying. What the console and MCP surface are the operational controls:

Action	Console	MCP
Pause a policy	✅ Schedule Explorer	pause_compaction_policy
Resume a policy	✅ Schedule Explorer	resume_compaction_policy
Run now	command shipped, button pending	run_compaction_policy
See declared policies	✅	list_compaction_policies
See run history	✅ Timeline	get_compaction_activity

Run now requires the compaction-policy.run capability and is written to the audit log. It is its own capability rather than sharing event-store.compact: that one is scoped to a single stream an operator named, while a policy run acts on every stream its selector matches — the same destructiveness, a much larger blast radius.

It runs the policy as declared. A dry-run policy dry-runs; there is no "armed just this once", because arming is a code change and an override would let a policy do something its declaration does not describe.

Pausing is the escape hatch: a policy compacting more aggressively than you intended can be stopped in one click while you decide what the threshold should have been.

Reporting

Every evaluation reports back to the console — one report per tenant scope, so an AllTenants policy reports each tenant separately rather than folding a tenant-specific failure into a fleet total. Each report carries the streams matched (the whole backlog, not just this tick's share), the number compacted, failures, exclusions, and a sample of stream identities.

Every run lands on the console Timeline under the Compaction category, so a compaction that ran last night sits next to everything else that happened to the service. A dry run files as info; an armed run files as warning, because it deleted events and this entry is the only lasting record of it.

A run that could not happen is reported too, as a CompactionRunFailed entry carrying the reason — a store that cannot compact, or a service without a CritterWatch licence. That last one matters more than it looks: compaction is a licence-gated operation, so on an unlicensed service a policy is declared, its schedule reads Running, and it never does anything. Without the entry there is nothing anywhere in the console saying why.

Each run is also written to the service's own log, at Information normally and Warning or Error when it could not run.

Declared policies themselves reach the console on the capability announcement, so a policy is visible before it has ever run — and because every field on it is code-first definition, changing a threshold or arming a policy re-announces on the next deploy rather than leaving the console showing the old values.

NOTE

The console's dedicated compaction-activity view and the operator controls above are still in progress (#1199). The declaration surface, the announcement, timeline entries and the live push are all in place.