Alert and notification settings

This page describes Darling v3.13.0. The source of this page is on GitHub at that version.

Audience: operators.

This page covers the alerts, analysis, smtp and webhooks blocks of darling.json, and the notification routes. It moved here from the Darling operator guide. For the alert types and the channels of both editions, see Alerts & Notifications.

alerts

The shared alert engine’s switches and thresholds. Every default mirrors Lite’s alert defaults exactly, so an empty section alerts like a fresh Lite install. enabled: false turns off all alert evaluation and analysis finding notifications (the analysis itself still runs and persists findings).

Key Default Meaning
enabled true Master switch for alert evaluation + finding notifications
cpuEnabled true
cpuThresholdPercent 80
cpuMode "total" "total" = SQL + other processes; "sql" = SQL process only
blockingEnabled true
blockingCountThreshold 3 Blocking waits (rolling window) that trip the alert on a SQL Server target, but only when the worst wait among them is also 15 seconds or more (a fixed rule, #7083). One blocked event of two seconds no longer pages; three events of 20 seconds do. The default was 1 before V235, which moves a store still at 1 to 3. One wait is one blocked session, blocker and wait start (#5644): the several blocked-process reports SQL Server writes for one long wait count once
pgBlockingCountThreshold 3 The PostgreSQL twin (#3444, V122): distinct ROOT blockers in the rolling window that trip Blocking Detected on a PostgreSQL target, floor 1. The same two rules apply as on SQL Server (#7083): at least this many roots AND a worst victim wait of 15 seconds or more, or one root whose worst victim waited blockingWaitSecondsThreshold. The default was 1 before V235. Its own key because the two count different things — engine-recorded blocked-process reports there, roots found in a periodic SAMPLE of pg_stat_activity here — so moving one deliberately does not move the other. blockingEnabled governs both engines. Inert on a store with no PostgreSQL targets
blockingWaitSecondsThreshold 60 Since #7083 (V235 moves a store still at 0) this one number is two bars. A single blocked event whose own wait reaches it pages Blocking Detected at once, whatever the count. Repeated reports of one blocked process count as one event at its largest wait. And it is the total blocked wait, in seconds, summed across the latest blocking snapshot, for the gate below; 0 = both off. A second gate beside the count one, because a count cannot tell one session blocked for an hour from one blocked for a second. Reports as its own “Blocking Wait Time” alert, and unlike the count gate it is level-triggered: it re-fires every cooldown while the wait stays above the threshold and clears when it drops below. Since #3653 the fire sits behind the shared persistence gate as a disjunction: one snapshot at or above 3× the threshold fires at once, anything between the threshold and that multiple must hold for 3 consecutive collected snapshots (counted per collection, not per 30 s sweep; a quiet collection between two breaching ones starts a new episode, because the collector writes rows only while something is blocked). The card’s Fired By field says which arm admitted it (single_snapshot_3x / consecutive_k3). Measured on one production store class: 97 of 102 episodes were a single snapshot, so a plain consecutive gate would have dropped exactly the severe pile-ups
deadlockEnabled true
deadlockCountThreshold 1 Deadlock count (rolling window) that trips the alert on a SQL Server target
pgDeadlockCountThreshold 1 The PostgreSQL twin (#3444, V122): distinct deadlocks in the rolling window that trip Deadlocks Detected on a PostgreSQL target, floor 1. Its own key because the SQL Server figure is calibrated against the fleet’s deadlock health band (health_bands.deadlock_warn_per_hour on get_alert_settings), a band a PostgreSQL server does not have. deadlockEnabled governs both engines. Inert on a store with no PostgreSQL targets A fire pages when the server had 5 or more distinct reports in the last hour or when a statement pair is new (not seen in the last 30 days). Otherwise the fire is held back from paging and recorded as one digest row for the daily digest. The pair is the sorted statements of the report’s graph, counted in the 30-day signature store under a pg: key. A report with no statements falls back to its lock shape. The alert text lists the statement pairs with their report counts.
poisonWaitEnabled true THREADPOOL / RESOURCE_SEMAPHORE / RESOURCE_SEMAPHORE_QUERY_COMPILE on SQL Server; IPC:BtreePage / IPC:BufferIo on Aurora PostgreSQL. One switch, one alert name, one shape on both engines: the alert fires when a poison wait type accumulates 600 s of wait inside a rolling ten-minute window — an average of one task (SQL Server) or backend (PostgreSQL) continuously stuck — as WARNING, and 6,000 s (ten stuck) as CRITICAL, judged per wait type. The bars are constants shared by both evaluators (PoisonWaitEvaluator), fleet-calibrated ~100× (SQL Server) and ~160× (PostgreSQL) above the worst healthy ten-minute bucket measured, and a mute rule on Poison Wait means the same thing whichever engine the server runs
poisonWaitThresholdMs 500 No longer consulted. Formerly the SQL Server per-wait average bar; retired by the shared accumulation shape above. Still read, written and reported (get_alert_settings) so existing configuration files and control-plane rows round-trip
runnableQueueEnabled true The Runnable Queue alert (#5641, V177): SQL Server only. Fires when the average of runnable tasks per scheduler (cpu_scheduler_stats.avg_runnable_tasks_count: the collections in the last 5 minutes when there are at least 3, otherwise the newest 3 if all fall in the last 10 minutes (or in 4 times the gap between cpu_scheduler_stats collections when that is longer, so the Low-Impact preset’s 5-minute collection still evaluates)) reaches the Warning or Critical bar below. Only Critical pages: a Warning is held as a digest row and sends nothing (#7086). Stored as config_alert_settings.runnable_queue_enabled; get_alert_settings reports it as runnable_queue.enabled and update_alert_settings writes it
runnableQueueWarningPerScheduler 8 Warning bar, runnable tasks per scheduler (the window average above). At least 1. Settings window, runnable_queue.warning_per_scheduler
runnableQueueCriticalPerScheduler 16 Critical bar, same unit. Never below the Warning bar: the engine raises a lower value to the Warning bar, and the Settings window and update_alert_settings refuse a pair in the wrong order rather than rewrite it. Settings window, runnable_queue.critical_per_scheduler
clientTimeoutsEnabled true The Client Timeouts alert (#5640, V179): SQL Server only. Fires when client attentions per minute reach the Warning or Critical bar below. Each bar is relative to the server’s own normal rate (#5736). An attention is the event that SQL Server records when a client cancels a query or its timeout expires. The rate is the attentions that the attention collector counted, divided by the minutes that its last runs cover (about 3 minutes). The alert is on by default since V216 (#5736): the upgrade turns it on for a store that had it off, because nothing offered off as a choice before then. Switch it off again in the Settings window if you do not want it. Stored as config_alert_settings.client_timeouts_enabled. get_alert_settings reports it as client_timeouts.enabled, and update_alert_settings writes it
clientTimeoutsWarningPerMinute 10 Warning FLOOR, attentions per minute (the rate above). At least 1. Since #5736 the Warning bar is the higher of this floor and clientTimeoutsWarningMultiple times the server’s own baseline. Settings window, client_timeouts.warning_per_minute
clientTimeoutsCriticalPerMinute 60 Critical FLOOR, same unit. Never below the Warning bar. The engine raises a lower value to the Warning bar. The Settings window and update_alert_settings refuse a pair in the wrong order rather than rewrite it. Settings window, client_timeouts.critical_per_minute
clientTimeoutsWarningMultiple 3 The Warning bar is at least this many times the server’s own baseline, the p99.9 of its 3-minute average attention rate over the last 7 days (#5736, V211). At least 1. Settings window, client_timeouts.warning_multiple
clientTimeoutsCriticalMultiple 10 The Critical bar is at least this many times the baseline. Never below the Warning multiple. Settings window, client_timeouts.critical_multiple
identityCapacityEnabled false The Identity Capacity alert (#6875): SQL Server only, off by default. Fires when an identity column has used the Warning or Critical share of its capacity (from its seed to the end its increment moves toward). Settings window checkbox, identity_capacity.enabled
identityCapacityWarningPercent 70 Warning bar, percent of the capacity used (1 to 100). A threshold override (identity_capacity.warning_percent) replaces it for one server or one database. update_alert_settings group identity_capacity
identityCapacityCriticalPercent 90 Critical bar, same unit. Never below the Warning bar: a lower value is raised to it. Override key identity_capacity.critical_percent
longRunningQueryEnabled true
longRunningQueryThresholdMinutes 30
tempDbSpaceEnabled true
tempDbSpaceThresholdPercent 80
lowDiskEnabled true Volume free space; graded CRITICAL when critically low
lowDiskThresholdPercent 10 Fire below X% free; 0 disables this dimension (clamped 0–100)
lowDiskThresholdGb 5 Fire below X GB free; 0 disables this dimension
longRunningJobEnabled true SQL Agent job running long vs. its history
longRunningJobMultiplier 3 Fires at 3x the job’s historical average
failedJobEnabled true Live msdb check for recently failed jobs
failedJobLookbackMinutes 60 Clamped 1–1440
agentAlertGapsEnabled false The “Agent Alert Gaps” alert (#6874): fires when a server’s SQL Server Agent alert setup has a gap (no alert for severities 19 to 25 or errors 823 to 825, an alert that tells no operator, a disabled alert, or a firing that told no one). SQL Server only. Off by default, like the other alerts that start firing on upgrade
backupOverdueEnabled false The “Backup Overdue” alert (#6873): fires when a database’s last full backup is older than backupOverdueThresholdHours. Off by default
logBackupOverdueEnabled false The “Log Backup Overdue” alert (#6873): fires when a database’s last log backup is older than logBackupOverdueThresholdMinutes. Off by default
checkDbOverdueEnabled false The “CHECKDB Overdue” alert (#6873): fires when a database’s last good CHECKDB is older than checkDbOverdueThresholdHours. Off by default
backupOverdueThresholdHours 168 Hours since the last full backup before it is overdue (7 days). A per-server or per-database threshold override on backup_overdue.threshold_hours replaces it (#7611)
logBackupOverdueThresholdMinutes 60 Minutes since the last log backup before it is overdue. The override is log_backup_overdue.threshold_minutes
checkDbOverdueThresholdHours 336 Hours since the last good CHECKDB before it is overdue (14 days). The override is checkdb_overdue.threshold_hours
cooldownMinutes 5 Minimum minutes between repeats of the same alert condition (clamped 1–120)
excludedDatabases [] Excluded from blocking/deadlock/long-running-query alert evaluation (collection unaffected)
excludedVolumes [] Mount points the Volume Free Space alert ignores on every server (#6098): the whole mount point, case-insensitive, a trailing \ or / on either side ignored (S:, S:\ and s:\ are one volume; /var/opt/mssql equals /var/opt/mssql/); never a prefix match. Dropped before the breach test, so the worst volume, the severity and the card come from the volumes that remain, and a server whose only low volumes are excluded resolves. Seeded into the store on first run (migration V200); the store row wins afterwards, edited in the Viewer’s Settings window, the web admin page or update_alert_settings. Collection is unaffected
longRunningQueryExcludedProgramNamePrefixes ["SQLAgent - TSQL JobStep"] Long-Running Query opt-out knob, program arm (#3653): sessions whose program_name starts with an entry are not evaluated by that alert (case-insensitive, no wildcards; applied ahead of the row cap). Seeded from a 7-day read of one large production store — SQL Agent job steps, ~460 sessions a week. A default, not a constant: [] excludes nothing by program
longRunningQueryExcludedLogins ["NT AUTHORITY\\SYSTEM", "NT AUTHORITY\\NETWORK SERVICE"] Long-Running Query opt-out knob, login arm (#3653): sessions whose login_name equals an entry are not evaluated (case-insensitive, whole name). Seeded with the two service principals that carried the read’s permanent multi-day background; the application’s admin login is deliberately not a default (it also runs real ad-hoc long-runners). [] excludes nothing by login

The long-running-query read shape is configurable too, on the same defaults Lite ships: longRunningQueryMaxResults (5, clamped 1–1000) and the five noise opt-outs longRunningQueryExcludeSpServerDiagnostics, longRunningQueryExcludeWaitFor, longRunningQueryExcludeBackups, longRunningQueryExcludeMiscWaits, longRunningQueryExcludeCdc (all true). Since #3653 the SQL Server read also carries the Long-Running Query opt-out knob: sessions whose program_name starts with a configured prefix or whose login_name equals a configured login are not evaluated by the alert at all — not read into the decision, not counted, not fingerprinted, the opposite of a mute rule — and the exclusion is applied inside the read ahead of longRunningQueryMaxResults, because the sessions it names are the longest-running on the server by construction and would otherwise fill the cap on every sweep. Its defaults come from a 7-day read of one large production store, whose long-running population was four classes: (1) SQL Agent job steps (SQLAgent - TSQL JobStep…, ~460 sessions a week across 11 jobs, medians 35–62 min) — the seeded program-name prefix; (2) the NT AUTHORITY\SYSTEM / NT AUTHORITY\NETWORK SERVICE logins (~70 sessions across 42 servers, medians 4.8–8.6 days, CDC-capture shaped) — the seeded exact logins; (3) the application’s admin login — deliberately not a default, because it carries the job wave but also real ad-hoc long-runners, and the job-step prefix already removes its share; (4) named humans — 3 a week, never excluded. Matching is case-insensitive with no wildcard grammar; the fired card’s Excluded Count item splits the sessions removed by arm (a session matching both arms counts once, under the prefix). On Darling the knob lives on the settings row since V135: two text[] columns on config_alert_settings (longRunningQueryExcludedProgramNamePrefixes / longRunningQueryExcludedLogins in darling.json, seeded on first run; the store row wins afterwards like every alert knob), edited in the Viewer’s Settings window or through update_alert_settings, reported by get_alert_settings; a pre-rung row reads as the seeds, and a list an operator clears stays cleared. Lite ships the same defaults in settings.json. The PostgreSQL twin consults the same knob through the same shared predicate builder (#3743): application_name is the program_name twin and usename (stored as username) the login_name twin, so an entry means the same thing on both engines — prefix for programs, whole name for logins, case-insensitive — applied ahead of the row cap, with the fired card’s Excluded Count split rendered the same way. The seeds name SQL Server internals and match nothing on PostgreSQL by construction; whether that engine wants seeds of its own is a production read, not a guess. On a PostgreSQL target the read also honours longRunningQueryExcludeBackups (it drops pg_dump / pg_dumpall / pg_restore / pg_basebackup sessions by application_name) and excludedDatabases through the same builder’s database arm, ahead of the same row cap (#3742) — the PostgreSQL twin applies all three exclusions in the read, so an excluded database’s sessions can no longer consume the page, and the fired card carries an Excluded By Database count beside the knob’s when the list is set — and unconditionally skips non-client backends (autovacuum workers, walsenders) and the VACUUM / ANALYZE / REINDEX / CLUSTER statements; the other three switches name SQL Server internals with no PostgreSQL reading and are ignored there — see the engine-coverage table below. Not configurable: the analysis-finding notification policy (notify at severity >= 1.5, 6-hour per-finding cooldown).

More alerts keys

These keys are in the same alerts section and are first-run seeds like the table above.

Key Default Meaning
notifyConnectionChanges true Send the Server Unreachable and Server Restored alerts
notifyConnectionDownAtStartup false Also announce a server that is already down on its first connect attempt after the service starts
connectionRefireMinutes 0 Re-announce a server that stays down every N minutes. 0 is off
notifyAgHealth true Master switch for the Availability Group alerts (failover, replica disconnect and reconnect, sync fell behind, database suspended)
agLagAlertSeconds 300 AG Sync Fell Behind fires when a secondary’s lag reaches this many seconds. 0 is off
agRedoQueueAlertKb 0 The second AG Sync Fell Behind trigger: a secondary’s redo queue in KB. 0 is off
agDisconnectRefireMinutes 0 Re-announce a replica that stays disconnected every N minutes. 0 is off
pvsEnabled true The persistent version store alert (SQL Server databases with accelerated database recovery)
pvsThresholdPercent 40 Fire when the version store reaches this percent of the database’s data files. 0 turns the check off
pvsFloorGb 1 A breach also needs at least this many GB of version store. 0 removes the floor
fileGrowthEnabled false The database file-growth alert. Off by default
fileGrowthRiseMb 10240 Fire when a database file grows faster than this many MB per hour, averaged over fileGrowthLookbackMinutes. 0 turns this gate off
fileGrowthVolumePercent 60 Also fire when a growing database file is over this percent of its volume. 0 turns this gate off
fileGrowthLookbackMinutes 60 The window the growth rate is averaged over. It does not rescale fileGrowthRiseMb
selfDiskFreeWarnPercent 10 Store Disk Pressure warns below this percent free on the store’s volume. 0 turns the check off
selfDiskFreeWarnGb 50 The percent also needs free space below this many GB, so a large volume at a low percent does not page. 0 removes the floor
collectionStaleMinutes 30 How long collection can go quiet before Collection Stopped fires
collectionFailureThreshold 10 Consecutive failures with no success that fire Collection Stopped without waiting out the window above
diskCriticalFreePercent 3 The Low Disk alert is graded Critical below this percent free
diskCriticalFreeGb 2 The Low Disk alert is graded Critical below this many GB free
analysisNotifyCooldownMinutes 360 Cooldown between notifications for the same analysis finding (clamped 30 to 10080)
storeJobCadenceWarnPercent 25 Store Job Over Cadence warns when a store job’s last run reaches this percent of its schedule interval. Critical is fixed at 100
retentionHoldWarnRatio 2.0 Retention Held warns when a policy held by rollup coverage is holding this many times its own horizon
retentionHoldCriticalRatio 4.0 The same for the Critical tier
fleetSweepEnabled true Run the scheduled fleet sweep. It has its own switch and keeps running when enabled is false
fleetSweepIntervalMinutes 60 How often the fleet sweep runs, 15 to 1440
dailyReportEnabled false Send the scheduled Daily Health Report (V232, #7041). Off until you switch it on. Like the fleet sweep it has its own switch and does not depend on enabled
dailyReportSendAt "07:00" The time of day the report goes out, written HH:mm in the report’s time zone
dailyReportTimeZone null The report’s time zone, an IANA id such as America/New_York. null or blank means the service host’s own time zone
dailyReportTitle null The report’s title. null or blank means Daily Health Report. At most 100 characters, no control characters
deadlockWarnPerHour 5 The deadlock health band’s Warning tier, in deadlocks per hour
deadlockCriticalPerHour 20 The deadlock health band’s Critical tier
blockingWarnPerHour 5 The blocking health band’s Warning tier, in blocked waits per hour over the window a card counted them in (#6927). Clamped on read
blockingCriticalPerHour 20 The blocking health band’s Critical tier, in blocked waits per hour
blockingWarnWaitSeconds 10 The blocking band’s Warning bar on the longest single block, in seconds
blockingCriticalWaitSeconds 60 The blocking band’s Critical bar on the longest single block, in seconds
blockingWarnSnapshotWaitSeconds 0 The blocking band’s Warning bar on the longest total blocked wait in one sampled snapshot, in seconds (#7028). 0 turns the tier off; valid values are 0 and 1 to 86400. When both tiers are above 0, critical is at least warn
blockingCriticalSnapshotWaitSeconds 0 The blocking band’s Critical bar on the longest total blocked wait in one sampled snapshot, in seconds. 0 turns the tier off
databaseStateEnabled true The database-state alert (a database in a state other than its expected one)
deliveryMode "Summary" How deadlock and blocking alerts are delivered: "Summary" is one batched card per cycle, "PerEvent" is one notification per incident. A server’s alertDeliveryModeOverride wins over it
perEventMax 5 In "PerEvent" mode, the most incidents sent one by one per cycle before a trailing “+N more” (clamped 1 to 100)

Which engine each built-in alert and health band covers

One alert name means one condition on both engines wherever both engines can honestly measure it; where only one engine can, the row says why rather than leaving the gap to be discovered (#3539). The health bands are the fleet card’s dots (get_fleet_overview, /api/fleet, the viewer’s Overview).

Condition SQL Server PostgreSQL Why one-sided, where it is
Deadlocks (alert) Extended-event graphs, deadlockCountThreshold per rolling window Server-log deadlocks parsed into pg_deadlocks, deadlocks.pg_count_threshold (V122) Both. Separate count knobs because the instruments differ (a captured graph vs a parsed log entry) and an operator tuning one should not silently move the other. The SQL Server fire is graded since #3653: WARNING by default, CRITICAL when the hour’s rate reaches the health band’s deadlock_critical_per_hour tier below — the count knob decides whether it fires, the band’s pair decides the colour, so the alert row and the fleet card agree about the same hour. A deadlock whose graph has a session from the monitor itself (SQL Server only) (application name starting PerformanceMonitorDarling or PerformanceMonitorLite) is left out of the count, though it is still stored and shown in the deadlock grids. The alert pages once per episode. An episode opens with the page and ends when the hour’s count falls below the threshold, which writes the Deadlocks Cleared notice. A count that rises inside the episode does not page again, except a rise to the CRITICAL rate (20 an hour). The next deadlock after a Cleared pages. The PostgreSQL host’s fire is graded the same way since #3653 — the same grader, the same one-hour count, the same store tiers — on its own deduped report count (the number the row quotes), not on the counter difference the band row below uses; the read caps at 50 distinct reports, so a Critical tier raised past 50/hr is unreachable on that host.
Deadlocks (health band) v_deadlocks graphs per hour pg_stat_database.deadlocks — the server’s own lifetime counter, differenced between consecutive samples per database, clamped at zero across a statistics reset, summed across the cluster’s databases Both, through the same health_bands.deadlock_warn_per_hour / deadlock_critical_per_hour tiers: a deadlock per hour is the same quantity whichever engine recorded it. The PostgreSQL arm bands only when at least one difference was taken in the window (a difference of fewer than two samples is not a zero), so a target nothing has collected from reads Unknown rather than Healthy. The fleet total’s deadlock_coverage.postgres_servers names how many servers were counted the counter way; get_pg_deadlocks has the graphs. Never SUM(deadlocks) on the store — the column is a lifetime counter repeated in every one-minute sample.
Blocking (alert) Blocking waits counted from blocked-process reports + DMV snapshots (one per blocked session, blocker and wait start), blockingCountThreshold (default 3, with a 15 second worst-wait floor), or one event at blockingWaitSecondsThreshold (default 60 s). The Cleared row says the alert clears when the last 60 minutes hold fewer than the count bar and no event at the wait bar, so it can arrive up to 60 minutes after the blocking ended. A count at the bar with every wait under 15 seconds pages no one. If no Blocking Detected incident is open on the server, one held digest row is written for the burst, and another only after the alert cooldown and only when a newer wait has arrived (“14 blocking events, worst wait 9 s, longest chain 2”). The daily digest lists it and the Alerts tab shows it as Digest. A held row stamps no page cooldown and records no escalation tier (it keeps its severity), so a real page right after goes out at once. A count bar of 0 and a muted server write nothing Distinct root blockers in a periodic sample of pg_stat_activity, blocking.pg_count_threshold (V122) Both; separate knobs because the denominators differ (engine-recorded reports vs sampled sightings). The PostgreSQL side holds the same way, worded “5 root blockers, worst wait 9 s, longest chain 2”. The PostgreSQL fire carries an explicit WARNING tier and no CRITICAL tier (#3653): the band row below borrows the SQL Server tiers in captures per hour, and no measured alert tier exists for sampled sightings, so a fire bar is not a tier.
Blocking (health band) Three arms, the #3596 tiers and the alerts’ own measures (#7028). Blocked waits per hour is the unit blocking.count_threshold counts in: health_bands.blocking_warn_per_hour and blocking_critical_per_hour, defaults 5 and 20. The longest single block is blocking_warn_wait_seconds and blocking_critical_wait_seconds, defaults 10 and 60. The longest total wait in one snapshot is blocking_warn_snapshot_wait_seconds and blocking_critical_snapshot_wait_seconds, both 0, which is off. The tiers keep their stored values. One wait is one count, however many report rows SQL Server wrote for it. Inside a window over an hour, each hour takes blocked-process report waits if it has any, else DMV snapshot waits, which is the blocking alerts’ own rule for their rolling hour. A window where some hours have only DMV snapshots can now count more than before, because those hours are no longer dropped. get_alert_settings reports blocking_band_unit: waits_per_hour for SQL Server and blocking_band_unit_postgres: captures_per_hour_at_1_minute_cadence for PostgreSQL. Lite stays on report rows because #7028 was built for Darling only: Lite’s tiers and text are unchanged, and there the 5 and 20 tiers count blocked-process reports per hour. The snapshot tiers read the same sum as the Blocking Wait Time alert, behind the same consecutive-sample gate. Set them to blocking.wait_threshold_seconds to colour a card on the measure that alert fires on Banded from sampled captures (#7616) The same two count tiers, health_bands.blocking_warn_per_hour and blocking_critical_per_hour, but the unit is blocked captures per hour at the default 1-minute cadence, not waits. PostgreSQL blocking is sampled once a minute, so a waiter seen in three consecutive captures is one wait seen three times. The card takes the share of its pg_blocking captures that saw any blocking, over the window it reads, and multiplies by 60. With the default 5 and 20 tiers, that is 5 and 20 minutes an hour with a blocked session in view. For PostgreSQL a tier above 60 means every capture, so a target blocked in every capture still reaches Critical. A card with fewer than 3 captures is Unknown, and a measured 0 reads 0. The two wait-seconds arms do not apply, because a capture shows who waits behind whom at one instant and not how long the wait has run. The Blocking alert still speaks for PostgreSQL separately, through blocking.pg_count_threshold.
Blocking Wait Time Sum of blocked wait across the latest live DMV snapshot, blockingWaitSecondsThreshold; since #3653 gated: 3× the bar on one snapshot, or 3 consecutive collections — SQL-only. The condition is a live total of how long sessions are currently blocked, which the DMV snapshot gives exactly. pg_blocking is a per-minute sample of pg_stat_activity: summing observed waits across one capture is a lower bound on the same quantity taken at an arbitrary instant, and a threshold in seconds against a once-a-minute sample would fire on the sampling grain as much as on the workload. A sampled twin is possible but needs its own stated denominator; not faked under the SQL Server name. Since #7028 the blocking health band has a third arm on this same sum, with the same gate. Its keys are health_bands.blocking_warn_snapshot_wait_seconds and blocking_critical_snapshot_wait_seconds. A 0 turns a tier off, and otherwise the value is 1 to 86400. When both tiers are on, critical is at or above warn.
Long-Running Query sys.dm_exec_requests, five noise opt-outs (sp_server_diagnostics, WAITFOR, backups, misc XE waits, CDC) + session_id > 50 + excludedDatabases + the #3653 opt-out knob (program_name prefixes / exact logins, seeded with the SQL Agent job-step prefix and the two NT AUTHORITY service logins, applied ahead of the row cap) Latest pg_session_states capture, same threshold and switch Both. PostgreSQL opt-outs: non-client backends (unconditional — mirrors session_id > 50), VACUUM/ANALYZE/REINDEX/CLUSTER by command tag (unconditional — SQL Server needs no statement-shape switch because its row carries the text and a mute rule can match it; pg_session_states stores no text, so the tag is the only handle), dump/restore utilities by application_name on the shared longRunningQueryExcludeBackups, the #3653 opt-out knob over application_name / usename through the same shared predicate builder and ahead of the same row cap (#3743; the SQL Server seeds match nothing here), excludedDatabases as the same CTE’s third flag ahead of the same cap (#3742). CREATE is reported (with its tag) because the tag cannot tell an index build from CREATE TABLE AS. The remaining three SQL Server switches name SQL Server internals; pg_sleep (WAITFOR’s cousin) is invisible without text. The PostgreSQL fire carries an explicit WARNING tier and no CRITICAL tier (#3653): the only bar the alert has is the operator’s threshold, and no measured duration distribution exists on either engine to cut a second tier from. On PostgreSQL (#7086) the page must come from a recent pg_session_states capture. Recent means no older than two cycles of that collector’s cadence (schedule overrides apply) or 10 minutes, whichever is larger. At the default 1-minute cadence a capture up to 10 minutes old still pages and an older one does not. A capture that is too old does not stamp the cooldown, so the first fresh capture that still shows the query pages. The detail text has a “Longest query” item with the pid, the minutes and the statement. The statement is looked up by query_id in pg_statement_text. mcp.dataDetail full prints it, maskLiterals prints it with literals masked, and metadataOnly leaves the statement out.
Poison Wait THREADPOOL / RESOURCE_SEMAPHORE / RESOURCE_SEMAPHORE_QUERY_COMPILE IPC:BtreePage / IPC:BufferIo (Aurora) Both, one shape since #3593: accumulated wait over a rolling ten-minute window, graded WARNING/CRITICAL from the shared PoisonWaitEvaluator bars on both engines. The earlier divergence (always-CRITICAL on SQL Server, graded on PostgreSQL, under one mute key) is closed. Both hosts clear a standing alert only on an observed window (#3593 SQL Server, #3653 PostgreSQL): a collector that stops delivering holds the alert open rather than announcing a recovery nobody measured. On PostgreSQL the witness is the pg_wait_stats collector’s own collection_log runs inside the window, because since #2694 a quiet poison event writes no row and the rows alone cannot tell a finished storm from a dead collector. On PostgreSQL a Warning that has not held for two windows writes one digest row and pages nothing.
Runnable Queue cpu_scheduler_stats.avg_runnable_tasks_count, the average over the collections in the last 5 minutes when there are at least 3, otherwise over the newest 3 if all fall in the last 10 minutes (or in 4 times the gap between cpu_scheduler_stats collections when that is longer, so the Low-Impact preset’s 5-minute collection still evaluates), WARNING at runnableQueueWarningPerScheduler (8), CRITICAL at runnableQueueCriticalPerScheduler (16) None SQL Server only: the signal is runnable tasks per scheduler, which the SQL Server scheduler collector stores and PostgreSQL has no counterpart for. A window with neither 3 collections in 5 minutes nor 3 inside the fallback window neither fires nor clears a standing alert, and a standing alert clears only on an observed quiet window or, after 30 minutes with no evaluated reading, resolves with that reason stated. The 30 minutes count from the last evaluated reading, and a window that is not an observation never restarts the count. Only CRITICAL pages. A WARNING is kept as a digest row and nothing is sent. The reason on the row is “inside a High CPU incident” when one is open on that server, and “Warning: pages at Critical only” otherwise. A CRITICAL right after a held WARNING pages at once (#7086). Across an upgrade, a Critical that follows a Warning the old version paged is still delivered past the cooldown. The email, webhook and viewer tray cooldowns let it through. Mute it with MetricName = "Runnable Queue" (no wait type)
Forced Plan Failing Query Store forced-plan failure counters, per plan (database, query id, plan id) None SQL Server only (#7086). The first failure of a plan (database, query id and plan id) is kept as a digest row and nothing is sent. A later collection pages when the failure count rises within 24 hours of the first failure. That covers a plan that keeps failing and a plan that fails again after it recovered. After 24 hours the plan starts over and is held again. A plan that was only held gets no “no longer failing” notice. The first-failure memory lives in the running process, so a restart holds a plan’s next failure once more. Mute it with MetricName = "Forced Plan Failing"
Client Timeouts attention_events rows from the attention collector. The bars are relative to each server’s own normal rate (#5736): WARNING at the higher of clientTimeoutsWarningPerMinute (the floor, 10) and clientTimeoutsWarningMultiple (3) times the server’s baseline, CRITICAL at the higher of clientTimeoutsCriticalPerMinute (the floor, 60) and clientTimeoutsCriticalMultiple (10) times the baseline. The baseline is the p99.9 of the server’s 3-minute average rate over the last 7 days, read about once a day, and a server with under a day of capture, or under 720 minutes (half a day) in which the collector ran, is graded on the floors alone. On by default None SQL Server only: the signal is an Extended Events session that PostgreSQL has no counterpart for. The alert is on by default. A Warning that worsens to Critical is delivered at once, past the cooldown. A mute rule with a client application pattern lowers the count in proportion to the share of the sampled events that the rule covers. The alert also leaves out two kinds of noise, in the rate and in the baseline (#7061). One is sampled events that lasted under 1 millisecond. They carry no statement text. They are the SqlClient driver cancelling a command that had already finished, which no user saw. A real client timeout lasts about as long as the command timeout, often 30 seconds, and an event with no recorded duration still counts. The other is Azure Data Studio’s own metadata queries: azdata running SMO column reads or create table #SVer. Its query editor uses the same azdata name, so the alert tells them apart by the shape of the statement. An azdata query of any other shape still counts. The exact count cannot be filtered by duration, so the rate is the exact count scaled by the share of sampled events that remain. Mute it with MetricName = "Client Timeouts"
Identity Capacity identity_capacity rows from the identity collector (every 6 hours, catalog views only, the top 20 columns above 10 percent used per database plus one marker row per scanned database). Each column is judged by its own percent used: WARNING from identityCapacityWarningPercent (70), CRITICAL from identityCapacityCriticalPercent (90), with the same band the get_identity_capacity tool, the fleet card chip and the Viewer tab show. The newest scan of each database in the last 7 days is judged, so a database the pass budget has not reached again keeps the answer of its last scan, and a server with no scan in that window is judged on nothing. The detail names the first 10 columns with their percent and days to exhaustion. One Identity Capacity Cleared history row is written when no column is at a bar any more. Off by default None SQL Server only: identity columns are a SQL Server catalog fact. The alert is off by default. A Warning that rises to Critical is delivered at once, past the cooldown, and a standing alert repeats after the connection refire cooldown. A mute rule with MetricName = "Identity Capacity" silences it
High CPU Ring buffer, 5-sample persistence gate, held open until CPU stays under the bar for 15 minutes (#7085) Performance Insights, banded on ACU headroom, its own 3-sample gate that resolves after 2 clear samples (#7082) Both. Both fires are graded since #3653: WARNING at the configured threshold, CRITICAL at the CPU health band’s 95% bar — the card’s ladder, not a new number. On PostgreSQL the grade is the band’s own classifier on the same two figures the card is handed, so the 95% is of the configured ACU ceiling (the figure the gate thresholded), never of the currently allocated capacity. On SQL Server a mute does not count as the page: a muted incident writes muted history rows, at most once per cooldown, and if the mute ends while the incident is still open it pages once within one cooldown of the last muted row, then only an escalation pages. On PostgreSQL a sample breaches only when instance CPU is also at or over the same threshold (#7082). Aurora Serverless v2 holds capacity for memory. Capacity at the ceiling with low instance CPU is not CPU load, so it does not page. The detail text shows both figures. On PostgreSQL, capacity at the ceiling with instance CPU under the bar writes a digest row at most once per cooldown and pages nothing, and a sample with no capacity figure counts on instance CPU alone only when the target is known to be provisioned (its oldest stored sample in the last 7 days is at least an hour old and none of them carries a capacity figure). On a serverless instance such a sample is left out of the gate.
Database File Growth Per-file growth over the lookback window vs FileGrowthRiseMb — SQL-only. It is a SQL Server file-model condition: data/log files with explicit growth increments and ceilings, where a rise means a file autogrew. PostgreSQL has no file growth to watch — relations grow a page at a time and the disk-level answer is Volume Free Space. The PostgreSQL relative is bloat growth (pg_table_bloat_stats hourly and pg_index_bloat daily are collected); a “bloat growth” alert is a future item and would not share this name, because it answers a different question.
Agent Alert Gaps The newest fresh agent_alerts capture (every 15 minutes), judged by the same status get_agent_alerts reports: no alert for severities 19 to 25 or errors 823 to 825, an enabled alert that tells no operator, a disabled alert, an alert that fired in the last 24 hours and told no one, or no alert defined at all — SQL Server only, off by default (get_alert_settings agent_alert_gaps.enabled, set it with update_alert_settings). A Warning that names the gaps, re-fires only on the connection re-fire interval, and writes one “Agent Alert Gaps Cleared” history row when the gaps are gone. Azure SQL Database, Azure SQL Managed Instance and Amazon RDS never capture, so it never fires there. A server whose captures stopped judges nothing: Collection Stopped reports that. The fleet card shows a chip for the same gaps whether or not this alert is on, and the band does not change.
TempDB Space, Volume Free Space, Long-Running / Failed Agent Job Yes (TempDB Space: 3-sample persistence gate) — SQL-only by construction (tempdb, host volumes via sys.dm_os_volume_stats, SQL Agent). TempDB Space sits behind the same shared persistence gate as High CPU since #3653 (A5): 3 consecutive collected tempdb_stats samples at or above the threshold to fire, the first collected sample below it to resolve, counted per collection rather than per 30 s sweep, and a reading that stops arriving freezes the gate rather than resolving it — measured on a 43-server production fleet, every one of one server’s 22 crossings in 14 days was 1–4 samples long. TempDB Space fires with an explicit WARNING tier and no CRITICAL tier (#3653): the product has no measured “tempdb nearly full” bar to cite, and the percent’s denominator is the growth ceiling where one exists and the allocation where not, so a single reserved-percent tier would mean two distances from failure.
PostgreSQL Wraparound Risk, PostgreSQL Vacuum Horizon Blocked — Yes PostgreSQL-only by construction: MVCC transaction-id wraparound and the xmin horizon have no SQL Server counterpart.
PostgreSQL Replication Slot Retention — pg_replication_slots retained WAL, wal_status The SQL Server analogue is a database whose log cannot truncate — sys.databases.log_reuse_wait_desc reading REPLICATION / AVAILABILITY_REPLICA / ACTIVE_TRANSACTION / LOG_BACKUP for consecutive collections. That column IS collected, but only as the load-time database_config snapshot (frequency 0), not as a time series, so a persistence-gated “Log Reuse Blocked” condition cannot be evaluated honestly today; it needs the column on a cadenced per-database collector (database_states is the natural home), which is a collector change with its own census and pins. Named here so the pair is on record: slot retention on PostgreSQL, log-reuse wait on SQL Server.
Server unreachable, Collection stopped, Retention Held, collector-health bands Yes Yes Both — these are about the monitor, not the engine.

Same-statement pileup is the one analysis finding evaluated on the COLLECTION cadence rather than the analysis interval, because the shape it describes does not survive a 30-minute wait: N concurrent sessions executing one statement, every copy past 10 seconds, at least one suspended on an IO-class wait, where that statement’s own snapshots over the prior 45 minutes were sub-second — the live signature of a shared parameterized plan flipping to an IO-heavy shape, which takes every caller down together and is usually over in about ninety seconds. It is computed from the active-query snapshot collector alone, so it works on deployments with the Query Store readers disabled, and it rides the notification policy above unchanged (severity >= 1.5 pages; the severity scales with concurrent sessions × elapsed, so a brief threshold pileup persists as a visible finding without paging). Episodes of the same statement share one incident id, so a recurrence folds onto the same trail as an occurrence rather than arriving as a fresh surprise — and, like every standing condition, it does not re-page inside the cooldown unless it worsens. Unlike the scheduled pass it has no 24-hour data-span requirement: its evidence is the snapshot plus that statement’s own recent history, so it works on a young install.

analysis

Scheduled analysis: the triage engine runs per server on a cadence, persists its findings to analysis_findings (the Recommendations surfaces, get_analysis_findings and the web dashboard read them the instant they land) and, independently, notifies on the ones that cross a severity floor. The first four keys are control-plane knobs (config_alert_settings, editable from the Viewer and update_alert_settings); the last is file-level.

Key Default Meaning
enabled true Run the scheduled analysis pass at all
intervalMinutes 30 Per-server cadence (clamped 5–360)
notificationsEnabled true Deliver findings to the channels at all (analysis still runs and persists with this off)
notifySeverity 1.5 The severity floor (0–2) a finding must reach to be notify-worthy
uncorroboratedRoute "digest" Where a notify-worthy but UNCORROBORATED finding goes (#3712). "digest" keeps it off email, Teams, Slack, PagerDuty and the generic webhook: it is still persisted, still visible on the web dashboard and MCP the moment it fires, recorded in the alert history as notification_type: digest with the gate’s reason (routing_reason on get_alert_history), and named once a day in the Analysis Singles Digest. "page" restores the pre-#3712 behaviour where every finding at or above notifySeverity pages. The file half of a two-home knob since V137: config_alert_settings.analysis_uncorroborated_route (nullable, CHECK digest/page) is the store half, and the running service resolves store non-NULL wins over file — a route set in the Viewer’s Settings → Automated Analysis combo or through update_alert_settings is live within one collection sweep and wins over this value; NULL there (every store after the V137 upgrade, and every fresh seed — the seed deliberately does not copy this value in) defers to this key, read once at service start; a value here that is neither spelling defers to the shipped digest. get_alert_settings reports the effective route and which home decided it (analysis.uncorroborated_route_source: store / file / default); sending analysis.uncorroborated_route: null to update_alert_settings clears the column so this key governs again.

Confidence chooses the channel. A finding earns a page by corroboration, never by severity alone: two or more facts in its story chain (each traversed edge is a fired predicate onto a fact that itself scored), or a matched co-fire check on its root fact (every check the scorer defines is a predicate over ANOTHER fact — a sibling anomaly in the same window, an absolute fact at its concerning bar, a wait that is significant), or one of the two by-construction stories (the absolution, the same-statement pileup). A lone fact — one fact, no matched check — is a single, and singles route to the digest under the default. The gate reads those components, not the confidence scalar, so a future re-weighting of the confidence formula cannot move the paging bar by itself. The extremity escape keeps its severity power (an extreme single still clears the floor and still leads its story), but its published intent — extreme and corroborated — is the enforced bar for a page. The measurement behind the default: the first day the recalibrated engine ran on a large production fleet it paged forty-plus times in seven hours on anomaly singles at severity 1.5–1.9 and confidence 0.20–0.35, interleaved with the handful that needed a human; most were true and redundant with a page something else had already sent.

Escalation is the corroboration event. A single that later gains a fact or a matched check pages at that moment as a NEW firing — the digest route cools in its own key namespace and digest rows are excluded from the page cooldown’s restart seed, so an earlier digest entry can never be the cooldown that eats the page. An uncorroborated single riding in an incident with a corroborated finding is named on that page as co-fired, exactly as before.

Recurring stories and one-fact stories at the bar go to the digest too (#7084). A story whose frozen text carries the recurring at this hour label (the same weekday and hour for three weeks in a row) routes to the digest with its severity and label unchanged, whatever else is true of it, except the absolution and same-statement pileup stories. A one-fact story whose severity is under the notify bar plus 0.25 (1.75 at the default 1.5) routes to the digest even with a matched co-fire check: one matched check alone no longer earns a page this close to the bar, while a second fact in the chain still pages at any severity. The reason is on the Alert History row and on get_alert_history as routing_reason, and uncorroboratedRoute set to "page" restores paging for both. An incident still pages if any member pages.

A corroborated finding still has to cost something to page (#7084). Three families paged on a statistical move alone, so each now needs a measured figure over a floor. A finding under its floor is still a finding: it is saved and shown on every surface as before, and it sends nothing. It also spends no cooldown, so the same story pages the day it clears the floor.

Finding Pages only when Where the figure comes from
Write latency (ANOMALY_WRITE_LATENCY) The window’s average write latency is 20 ms or more. The finding’s avg_latency_ms. 20 ms is the write bar the detector already used on the window’s peak, now applied to the average too.
Plan-regression chain (any story with PLAN_REGRESSION on its path) The server’s average CPU in the window is 50% or more, or queries’ timeouts rose. avg_total_cpu of the CPU fact, and timeout_offender_count on the PLAN_REGRESSION fact (Query Store Aborted runs, which a client timeout produces). No CPU fact counts as 0%.
PostgreSQL wait profile (ANOMALY_PG_WAIT_PROFILE, ANOMALY_PG_SAMPLED_WAIT_PROFILE) The window’s mean wait rate is 1.0 or more waiting backends. Also no page for the same story went out on that server in the last 24 hours. mean_ms_per_sec divided by 1000. 1.0 is Poison Wait’s Warning bar. The 24-hour clock lives in memory. After a restart it is read back from the alert history’s last delivered page of that story, so a restart does not open a second page.

The Daily Digest is the third daily document beside the Collector Cost Digest and the Fleet Sweep Rollup. It is where an alert that was held back from paging reaches a person. Its metric name is Analysis Singles Digest. Notification routes, mute rules and webhook automations match on that name, so it stays as it was before the digest also carried held alerts. Its title on email, Slack and Teams is “Daily Digest”.

The digest goes out through the reports family at INFO, once per trailing day, whenever alerts are on. A delivered-today stamp in the store keeps it to one per day, and a day with nothing to report posts nothing.

An alert can be held back instead of paged: a first failure that has not repeated, or a small event below the paging bar. A held alert is recorded in Alert History with the status Digest and the reason on the row (routing_reason). Nothing is sent. The digest has two sections.

  • Held back from paging. One line per server and metric. It gives how many were held, the first and last time (UTC), and the reason stored with the newest one. Example: Prod-1 - Runnable Queue Warning: 3 held, first 2026-11-01 02:10 UTC, last 2026-11-01 09:40 UTC. <reason>.
  • Analysis singles. The top movers by severity, then a breakdown by family and server. Each line carries the alert-history key (Analysis: <category> [<hash>]) and the #1140 dedup fingerprints of the finding. Use them to open the full drill-down.

The short message counts both, for example “12 held alerts and 30 analysis singles across 9 servers”. A day with only one kind shows only that section. A condition that later meets its paging bar pages at that moment as its own alert, so nothing in the digest waits on it.

A mute rule wins over a hold: a muted alert is recorded as muted and is not in the digest. To keep one metric out of the digest, mute it. To change where the digest goes, edit the route for the reports family. Lite has no daily digest, and its Daily Summary leaves held alerts out of its count.

The Daily Health Report shows the held part of a metric’s count, as in Blocking Detected x12 (9 held). A metric with no held alerts reads as before.

Daily Health Report

The Daily Health Report (#7041) is one report a day on how every monitored server did in the last 24 hours. It is off by default. It goes out through the same delivery path as every other alert, as the Daily Health Report metric in the reports family, so the channels you already configured carry it. Email gets an HTML report. Slack, Teams and the generic webhook get a short text version.

The settings. The same four settings are edited in the Viewer (Settings, Notifications, Daily Health Report), on the web dashboard (Admin, Settings, for a signed-in admin) and over MCP.

Setting darling.json (alerts) MCP group (get_alert_settings, update_alert_settings) Store column (config.config_alert_settings)
Send a daily health report dailyReportEnabled daily_report.enabled daily_report_enabled (boolean, default false)
Send at dailyReportSendAt daily_report.send_at ("HH:mm") daily_report_send_at (time of day, default 07:00)
Time zone dailyReportTimeZone daily_report.time_zone (null = the host’s zone) daily_report_time_zone (text, NULL = the host’s zone)
Title dailyReportTitle daily_report.title (null = Daily Health Report) daily_report_title (text, NULL = the default title)

The columns arrive with store schema V232. A time that is not HH:mm, a zone the machine does not know and a title that is blank, over 100 characters or holds a control character are refused, and nothing is written. A refusal names the field. update_alert_settings and the web editor check a zone against the service host. The Viewer checks it against the machine it runs on, and a stored id the service host does not know is read as the host’s own zone.

What it contains. A fleet header, then each server worst first. Each server shows the day’s health band and its reasons. It then lists the alerts that fired, failed Agent jobs, databases past the full backup, log backup or CHECKDB bar, low disk space and collection problems. A healthy server takes one line. The report is built only from data the store already holds, so it adds no load on a monitored server. PostgreSQL targets show the sections that apply to them: failed jobs, database protection and low disk are SQL Server only.

  • The band is the same 24-hour band the fleet overview card shows for the server, not a second grading.
  • Collection problems and database protection are read as the store holds them when the report is built. They are not rebuilt as they stood at the end of the 24 hours. For the scheduled report that is the same moment. For an earlier day read with get_daily_health_report, they are today’s.
  • The alert count leaves out resolutions and the report’s own delivery row, so sending the report never adds to the count of alerts it reports.
  • Length. The chat and webhook text is short, at most about 2800 characters, and ends with “and N more servers” when a large fleet does not fit. As a last guard the delivery cuts any text over 10000 characters at a line break and points at the email and the web page. Teams rejects a card near 28 KB. The email HTML is never cut.

get_daily_health_report previews the report for any finished day, and send_daily_health_report_now sends a copy now (see MCP configuration).

When it is sent. The service checks on its normal alert pass, every 15 seconds. Until a report is due, a check costs one settings read.

  • Due rule. The report is due when it is switched on, the alert master switch is on, and the local time of day in the report’s time zone is at or past the send time. The first check after that sends the report for the 24 hours that end at the send time, even when the check runs late.
  • Once a day. It goes out at most once per local day. The service records each delivery in the store, so a restart does not send it again. If the store cannot answer, the service falls back on its own memory and can send once more after a restart.
  • Down at the send time. The report goes out on the first check after the service is back, on that same local day. A day that has passed entirely is skipped, never sent late.
  • Retry. A build that fails, or a send that fails on every channel, records nothing and is tried again 5 minutes later the same day. A send that was muted, or that had no channel to go to, counts as the day’s delivery.
  • Daylight saving. A send time inside a spring-forward gap is due at the first local minute after the gap. A fall-back repeated hour is one local day, so the report sends once.

Sending it now. send_daily_health_report_now builds the report for the 24 hours that ended at the most recent send time and sends it through the same path. It does not read or write the once-a-day record, so the scheduled report still goes out at its time and a recipient can get two. It works while dailyReportEnabled is off, and it is refused while the alert master switch is off. The tool needs mcp.allowWrites or an operator token in mcp.tokens.

Sending it to chosen recipients. Add a notification route (see Notification routes) that matches the reports family, or the Daily Health Report metric by itself, and give it the destinations you want. With a route for the metric the report goes to those destinations only for each channel the route sets. For example, a route for the metric with its own email recipients sends the report to those people, and the other reports documents stay on the parent channels.

smtp

Email delivery is enabled when host and from are set — there is no separate enable flag. to is the default recipient list: an alert that no notification route covers goes there, so with to blank such an alert sends no email (the recipients of a covering route are used instead).

Key Default Notes
host ""
port 587
useSsl true
username (none) For authenticated relays
encryptedPassword (none) Same --encrypt-password DPAPI pattern as SQL auth
password (none) A literal or an env:NAME / file:/path reference (#1804) — the non-Windows email path
from ""
to "" Comma-separated default recipients, for alerts no notification route covers; may be blank when routes name the recipients
emailCooldownMinutes 15 Email/webhook channel cooldown (clamped 1–120)
emailSendResolutions false Also send the email when an alert resolves

webhooks

A channel is enabled by a non-empty URL.

Key Default Notes
teamsUrl "" Teams incoming webhook
teamsProxy "" Optional proxy address
slackUrl "" Slack incoming webhook
slackProxy "" Optional proxy address
slackSendResolutions false Also post to Slack when an alert resolves
teamsSendResolutions false Also post to Teams when an alert resolves
genericUrl "" Generic webhook endpoint, stored sealed
genericHeaders "" JSON object of request headers, stored sealed
genericBodyTemplate "" JSON body with {{metric}} and {{server}} placeholders, or empty for the shared default
genericProxy "" Optional proxy address
genericSendResolutions false Also post to the generic webhook when an alert resolves
pagerDutyRoutingKey "" PagerDuty Events API v2 routing key, stored sealed
pagerDutyUseEuRegion false Use the EU PagerDuty endpoint
pagerDutyProxy "" Optional proxy address
pagerDutyAutoResolve false Close the PagerDuty incident when the alert resolves

Webhook and heartbeat posts do not connect to a link-local address (169.254.0.0/16, fe80::/10) or to the cloud instance addresses fd00:ec2::254, 168.63.129.16 and 100.100.100.200; loopback and private addresses stay allowed. The address check applies to direct connections: when a channel posts through a proxy, Darling checks the address the URL itself spells out when it is saved (a trailing dot and the IPv4-mapped, NAT64 and IPv4-compatible forms included) but cannot see what the proxy connects to, so the proxy decides what it can reach.

Generic webhook: status, one key per alert, and incident.io (#6718)

The generic webhook posts the JSON in genericBodyTemplate to genericUrl, with genericHeaders added. Every {{token}} in the template is replaced as the body is built, and text tokens are JSON-escaped. The tokens are {{metric}}, {{server}}, {{value}}, {{threshold}}, {{severity}}, {{context}}, {{detail}}, {{timestamp}}, {{status}}, {{dedup_key}}, {{resource_name}}, {{database}}, {{triage_url}}, {{context_json}}, {{incidents_json}} and {{investigation_json}}. The last three are raw JSON, so use them unquoted.

  • {{status}} is firing on an alert and resolved on a resolution. The Settings window’s test send says firing. Resolutions go out only when genericSendResolutions is on (see “Send resolutions” below).
  • {{dedup_key}} is the same for an alert and its resolution. A resolution renders the key the alert’s own post rendered, so a receiver that matches on a key can close what the alert opened. The key is the first incident’s own key, or <server id>:<metric> when the alert has no incident. A restart between the alert and its resolution does not change this.
  • {{metric}} on a resolution is the resolution’s title, such as CPU Resolved, and not the alert’s name. A template that needs the alert’s name on a resolution can build it from {{server}} and {{dedup_key}}. The resolution’s {{detail}} text also names the alert (Alert: ...).
  • {{value}} on a resolution is the word resolved, and {{threshold}} is empty.
  • A recovery notice that is its own alert, such as “Server Restored” or “AG Replica Reconnected”, is sent as an alert and not as a resolution, so its {{status}} is firing and its key is its own.

Example: open and close an alert in incident.io. Create an HTTP alert source in incident.io. It gives you a source config ID and a token.

Key Value
genericUrl https://api.incident.io/v2/alert_events/http/<ALERT_SOURCE_CONFIG_ID>
genericHeaders {"Authorization": "Bearer <ALERT_SOURCE_TOKEN>"}
genericSendResolutions true. Without it incident.io never gets a resolved event and the alert stays open.
{
  "title": "{{metric}} on {{server}}",
  "status": "{{status}}",
  "deduplication_key": "{{dedup_key}}",
  "description": "{{detail}}",
  "source_url": "{{triage_url}}",
  "metadata": {
    "severity": "{{severity}}",
    "server": "{{server}}",
    "database": "{{database}}",
    "resource": "{{resource_name}}",
    "context": {{context_json}}
  }
}

Use that as genericBodyTemplate. incident.io’s alert event takes title, status (firing or resolved), deduplication_key, description, source_url and metadata, and has no priority field. To set a priority, see the next paragraph.

Set the alert’s priority from {{severity}}. {{severity}} is one of Darling’s own words: CRITICAL, WARNING, INFO or ALERT. incident.io maps it to a priority. On the alert source, set Priority to a dynamic value and add a Query expression on Payload. Give it a step that extracts fields from JSON:

$.metadata.severity == "CRITICAL" ? "<URGENT_PRIORITY_ID>" : "<OTHER_PRIORITY_ID>"

Parse the result into Alert Priority. Use each priority’s ID or slug, not its display name, because a display name parses to nothing. The editor’s preview shows the priority each value resolves to.

What to know about incident.io:

  • incident.io allows 120 events a minute for each alert source. A request over that gets a 429 and incident.io drops the event. Darling makes one attempt and does not retry a 429, so the channel records a failed send.
  • Tested on 2026-10-24 against one incident.io HTTP alert source, with the template above and Darling’s own alert and resolution sends:
    • The alert and its resolution carried the same {{dedup_key}}, and the resolution closed the alert.
    • The alert kept the title from its first event. The resolution’s title did not replace it.
    • A key with spaces and colons, such as 3:High CPU, was accepted.
    • An empty source_url was accepted. {{triage_url}} is empty when Darling has no triage link, and the alert then has no link.
    • Nested objects in metadata were accepted. Whether a Priority or attribute expression can read a nested path was not tested.
    • A resolved event with a key that matches no open alert got a 202 and changed nothing.
    • A new alert with the same key after a resolution opened a new alert in incident.io.
    • An event with no status got a 422.
    • With no Priority expression, the alerts got a default priority. Add the expression if your alert routes use priority.
  • Not confirmed: any limit on the key’s length, and the largest body incident.io accepts.

Another receiver that matches on a key and takes a status works the same way. Darling has no native incident.io channel.

Notification routes — sending alert families to different channels

The channels above (and the SMTP, PagerDuty and generic-webhook ones the Viewer’s Settings window adds) are the parent channel set: one destination per channel type, and by default every alert goes to every configured one. One stream then interleaves four audiences — the monitor’s own health, scheduled reports, Agent job failures, and the performance pages the channel exists for — and the reader routes in their head on every post. Since V131 (#3598) a sparse routes table layered over the parent row lets the operator make that call once: Settings → Notifications → Manage Notification Routes…, or read it back with get_notification_routes.

A route names what it matches and a destination per channel type, and an empty channel on a route is passed on: to the next less specific matching route that sets it, and to the parent’s when none does — so a route can point one family’s Slack at a second webhook and leave PagerDuty exactly as the parent has it, or put a PagerDuty key on the performance route alone with none on the parent, which is how “only pages page” is spelled. What a route matches is a family, Analysis: <category> (every analysis finding of that category, whatever its hash), Analysis: * (every analysis finding, of any category, at any scope) or one exact metric name (as metric_name reads in Alert History; an exact route also catches that alert’s recovery — Server Restored routes as Server Unreachable, AG Replica Reconnected as AG Replica Disconnected). The four families are a closed taxonomy the product writes down (AlertFamily), pinned by a census so a new alert cannot ship unclassified:

Family Owns
self-monitor The monitor about itself: Collection Stopped, Capture Down, Compression Job Stuck, Refresh Job Stuck, Retention Job Stuck, Daily Rollup Stale, Store Job Failing, Store Disk Pressure, Store Autovacuum Behind (checked once an hour: a table with over 1,000,000 live rows has had more than 3 times its autoanalyze threshold in changes since its last ANALYZE, on two reads in a row. The alert names how many tables, the worst one, the autovacuum workers in use against autovacuum_max_workers and the longest worker. Fix it by raising autovacuum_vacuum_cost_limit and autovacuum_max_workers, with ALTER SYSTEM or a line below the include line of postgresql.conf on a managed store. A restart resets the two-read count, so after a restart the alert waits for two fresh reads), Store Runtime Upgrade, Store Job Over Cadence, Retention Held, Custom Alert Rules Unhealthy, Stale Mute Rules, Web TLS Certificate Expiring, MCP TLS Certificate Expiring, Store Settings Need Attention, Collector Cost Regression, Collection Falling Behind, Collection Gap At Start (once per start, when the service was not collecting for 15 minutes or more before it started), Collection Gaps In History (once a day, after 03:00Z, for the previous UTC day: an hour whose servers, or perfmon collection passes, fell below half of the usual for that hour; changing the perfmon collection schedule can trigger it for a few days)
reports Scheduled prose, never a page: Collector Cost Digest, Fleet Sweep Rollup, Analysis Singles Digest (titled Daily Digest), Daily Health Report
agent-jobs Failed Agent Job, Long-Running Job, Agent Not Running, Agent Alert Gaps
performance Everything about a monitored server the on-call is paged for: High CPU, Blocking Detected, Blocking Wait Time, Deadlocks Detected, Poison Wait, Runnable Queue, Client Timeouts, Identity Capacity, Long-Running Query, tempdb Space, Volume Free Space, Version Store (PVS), Database File Growth, Database State, Forced Plan Failing, the three PostgreSQL outage predictors, the five AG … alerts, Server Unreachable / Server Restored, every Custom:<id> rule and every Analysis: … finding — and any metric the taxonomy does not name, so an unclassified alert lands where every alert landed before routes existed
external Alerts another service raised through its own database login, named External:<source>:<key>. With no route they go to the parent channels like any other alert.

Resolution is per channel (scoped routes add a first level, described below): an enabled route matching the metric exactly, then one matching an analysis finding’s category, then one matching every analysis finding (Analysis: *), then one matching its family, then the parent — and within a level the lowest route_id with a non-empty column. Routing sits after the delivery cooldown and the per-metric repeat budget, so it changes where a post lands and never whether it is sent; a store with zero routes posts exactly what it posted before, byte for byte. Every delivered alert-history row records where it went (route on get_alert_history: the family, the matched route_id, and each delivered channel with the route that supplied it), so a misrouted post can be traced. What a route does not do: an empty column inherits, so a route cannot silence a channel the parent has — put the channel on a route instead of the parent, as above; proxies, the generic channel’s headers/template and the PagerDuty region stay on the parent (they say how a channel type is reached, not where a family lands); and the email column only names recipients, in place of the default to list or with it blank — SMTP host, from-address and credentials live on the parent. The pre-routes workaround — a generic-webhook payload template branching on {{metric}} in an external router — still works; it just is no longer the only way.

Example: analysis findings to their own channel. Findings are named Analysis: <category> [<hash>] and belong to the performance family, so by default they go wherever your threshold alerts go. To send every finding to a channel of its own, add one route with the match Analysis: * (pick *Every analysis finding (Analysis: ) in the route dialog) and a Slack URL, and leave everything else as it is. Threshold alerts such as High CPU and Blocking Detected still match the performance route or the parent, and a finding still goes to a route for its own category if you made one. The match works at any scope: add the same route on one server or one tag to send only that scope’s findings elsewhere. An Analysis: * route does not catch the Daily Digest (Analysis Singles Digest, a reports metric). Findings are always Warning, so do not set such a route to critical only: it would match nothing. A server or tag route still comes first for its own servers, so a scoped performance route keeps that scope’s findings on the channels it sets unless you add Analysis: * at the same scope.

Scoping a route to a server or a tag (V181, #5750)

A route can apply to all servers (the default), to one server, or to one tag. A tag route covers every server that has the tag or any of its sub-tags. A route has one scope at most.

You pick the scope in the Applies to box of the route dialog. The server box shows the display name, and the tag box shows the tag path, for example Production / East. With a scope you can also pick Every metric (*). That route catches every alert for its scope that no more specific route of the same scope matches. A route for all servers cannot use *.

The most specific route wins, in this order:

  1. The scope: a server route first, then a tag route, then a route for all servers.
  2. The metric: the exact name, then the paired firing (a recovery follows its alert), then the finding’s category (Analysis: <category>), then every analysis finding (Analysis: *), then the family, then *.
  3. For tag routes at the same metric level, the deeper tag wins. Deeper means more parent tags.
  4. If two routes still tie, the lowest route id wins.

A scoped route replaces the default for each channel it sets, as every route does. A channel that the route leaves empty still goes to the next route that sets it, or to the Notifications channel. An alert never goes to two destinations on one channel. The generic webhook headers and the body template stay app-wide. A scoped route changes only where the post goes.

If the server or the tag of a route is removed later, the route is orphaned. An orphaned route matches nothing, so its alerts go to the next route or to the default. The Notification Routes window shows Orphaned: server removed or Orphaned: tag deleted in the Status column, and get_notification_routes returns orphaned. The service never deletes an orphaned route for you. Delete it, or re-scope it, in the window. If the viewer cannot read the tag data, a tag route shows Not active: tag data unreadable and not OK.

A bring-your-own store needs one step after the service upgrades the store to V181. Run tools/provision-roles.sql again, so the viewer and mcp roles can read the two new columns. The compose store and managed provisioning do this on their own.

A bring-your-own store that was provisioned before 3.12 needs the same step for the MCP route, signing key and default channel writes. 3.12 changed the store functions behind create_notification_route, update_notification_route, generate_webhook_signing_key, remove_webhook_signing_key and update_notification_channels, and only tools/provision-roles.sql creates them. Run it again after upgrading. Until then each of those tools answers refused and tells you to re-run the script, and nothing is saved. The compose store and managed provisioning do this on their own.

Destinations are bearer secrets, so the viewer and mcp roles never read a webhook URL or key: the table’s generated configured_channels column says which channels a route sets, and the MCP surface administers routes in full: create_notification_route makes a route (with no destination of its own, each channel goes where it went before, to the next matching route and then to the channels configured in Settings), update_notification_route changes a route’s match, scope, enabled flag, resolution override and fallback, and both set a route’s own destinations (a Teams, Slack or generic webhook URL, a PagerDuty routing key, email recipients; - removes one). The service seals each webhook value for the route’s own row, in the transaction that creates or edits the route, exactly as the Viewer does, so a value copied to another route does not open there; an answer never shows a webhook URL or key: it says whether one is saved (with the host for Teams and Slack only), it shows the recipient list in full, and the undo of a destination change carries the earlier recipient list and says to pass an earlier webhook URL or key again. set_notification_route_enabled, set_notification_route_fallback and delete_notification_route change the rest. A route with an explicit agent setting (Off included) keeps its match and scope in the Viewer, and an agent route is changed only in the Viewer. The mcp role has no INSERT on the table, no write on the destination columns and no read of the webhook and key columns (it reads the recipient list and configured_channels); the route tools write through config.create_notification_route, config.update_notification_route and config.set_notification_route_destinations, functions that store only a sealed value or an empty one for a webhook and refuse an agent route. config.set_notification_route_destinations hands back only whether a webhook or key was saved before, for the columns the call names, and never a stored value; for a changed webhook value it accepts only the sealed form (sealed:v1:, the key id and base64, at most 16384 characters), and for a changed recipient list it refuses control characters, angle brackets, double quotes and more than 50 entries. Every writer (the Viewer’s settings and route windows, the MCP route tools) refuses a Teams, Slack or generic webhook URL at save unless it is an absolute http or https URL with a host and no user name or password and whose host is not a link-local or cloud instance address written out, and refuses an email recipient list that is not comma separated plain addresses (the function the mcp role writes through checks only the shapes listed above, and the service checks the rest before it calls it), and a direct webhook post never connects to a link-local address (169.254.0.0/16, fe80::/10), whether it came from the URL, a DNS answer or a redirect (a channel that posts through a proxy is only checked against the URL’s own address at save, because Darling cannot see what the proxy connects to); loopback and private addresses stay allowed for internal receivers. Routes hot-reload through the same config_version beacon as the parent row (the V131 trigger), so a change lands on the next firing with no restart.

Limiting a route to Critical or Warning alerts (V229, #6928)

A route takes alerts of both severities unless you limit it. The severities setting of a route is both (the default, and what every route you made earlier has), critical or warning. A route set to critical takes Critical alerts only. A route set to warning takes Warning alerts only. With one route per severity you can send production warnings to a team channel and keep the on-call channel for Critical alerts.

A route whose setting leaves an alert out does not match it. Matching goes on to the next route in the order above, and to the default channels when no route is left. It is the same fall-through as a route that is disabled or that names a different server.

  • Which severity an alert has. The alert’s own severity decides when it carries one. An alert that is graded when it fires, such as an external alert or a custom alert rule, carries the severity it fired at. Otherwise the alert’s badge decides: a red badge is Critical, and every other badge (Warning, Alert, Info) counts as Warning. So a Warning-only route also takes alerts that are not Critical. The exception is a recovery that is delivered as an alert of its own (Server Restored, AG Replica Reconnected): it is routed at the severity its firing was delivered at, so it takes the route the firing took, and the channel that saw the fire sees the clear.
  • A Warning alert that becomes Critical while it is open is delivered again as a new Critical alert. The Critical route takes it from then on.
  • A resolution goes to the channels that the alert’s fire reached. The severity setting is not read again when the alert clears. Editing a route between the fire and the clear does not move the clear. The PagerDuty close for an external alert uses the severity that the fire went out at. It reaches the routing key of the route that opened the incident.
  • Agent requests follow the same filter: a Warning alert does not open an investigation under a route limited to critical.
  • Set it in the route dialog of the Viewer or in the Routes list of the web dashboard. In MCP, use the severities argument of create_notification_route and update_notification_route. The value is critical, warning or both, in any case. Any other value is refused as invalid. get_notification_routes returns severities for every route.
  • Lite does not have this setting. A route you make in Lite always takes both severities.

Report notices: telling a channel when an agent files a report (V233, #6979)

A route can post a short notice to Slack or Teams when an agent files a report on one of its requests. Without this setting, the report is only on the Investigations page and in MCP. The report notice setting of a route is off, needs_person_or_change or all.

  • off is the default. Every route you made earlier has it, and so does every new route. Nothing changes when you upgrade. A route posts report notices only after someone chooses another value.
  • needs_person_or_change posts a notice for a report with the needs_person verdict. It also posts one for a report whose request has at least one agent change.
  • all posts a notice for every report.
  • Where it goes. Slack and Teams only. Email, PagerDuty and the generic webhook get no report notice. The notice goes to the Slack and Teams destinations of the route that opened the request. If the route leaves a channel empty, the notice uses the channel from Settings > Notifications, as an alert does. A channel with no destination at either place gets nothing. A report on a route with no destination at all is marked as handled. A destination that you add later sends no notice for it.
  • What the message holds. The verdict, the server, the metric and the first line of the root cause. It lists each agent change with its tool, its object and its result. It lists at most 10 changes and then adds a line that counts the rest. Undo is not in the message. Undo stays a person’s action on the Agent changes tab.
  • Links. The message links to the request, and to the Agent changes tab when the request has changes. Both links start from web.publicBaseUrl. If that setting is empty, or the web dashboard is off, the message has no links.
  • How often. A notice goes out at most once for each request. Darling writes a mark for the request before it sends, so a send that fails is logged and is not tried again. When you switch a route on, Darling sends notices only for reports filed in the last 30 minutes. Older reports stay silent, so a route that you switch to all does not post your whole backlog.
  • Data detail. The text follows mcp.dataDetail, as the reports do. With maskLiterals, the root cause and the object of each change have their literal values replaced. An apostrophe right after a letter or digit, as in session's or sessions' locks, stays as written. So a quoted value written right after a word with no space, as in THEN'alice', keeps that value. A value after =, as in name='alice', or after a one-letter prefix such as N'alice', is still masked. A possessive or a contraction does not mask the rest of the line. A -- or /* in the text is plain writing, not a comment, and a quoted value after it is still replaced. With metadataOnly, the message leaves out the root cause.
  • Set it in the Report notice box of the route dialog in the Viewer, or in the Routes list of the web dashboard.
  • Lite does not have this setting and sends no report notice. Lite has no agent changes and no event delivery.

Proposal notices (#7109)

When an agent proposes a server action on a route that needs approval, Darling sends one notice about the proposal. A person can then approve it without watching the request.

  • Where it goes. To the email, Teams and Slack webhook destinations of the route that opened the request. A route with no bot still sends the notice. A route with no destination is marked as handled and sends nothing.
  • What the message holds. The kind of action, the server and a link. It holds no statement, no reason and no target. When nobody can approve the proposal, it adds “Nobody can approve this. Map a person, or turn on single-person approval.”
  • The link. It is <web.publicBaseUrl>/#/proposals?proposal=<id>. With no web.publicBaseUrl, or with the dashboard off, the message says “Open the proposal in the web or the Viewer.”
  • How often. A notice goes out at most once for each proposal, and a retry by the agent does not send another. Darling writes a mark before it sends, so a send that fails is logged and is not tried again. It only looks at proposals made in the last 30 minutes.

For who can approve and how, see Level 3 server actions: approval.

Send resolutions (#5825)

A resolution is the message that an alert has cleared: the condition that fired it is no longer there. Each channel (Slack, Teams, the generic webhook and email) has a Send resolutions box in the Viewer’s Settings > Notifications, off by default. When it is on, a resolution goes only to the destinations of that channel that the alert’s fire reached; an alert whose fire was not sent gets no resolution. A service restart between the fire and the clear does not lose this: the clear reads the fire’s alert-history row, which names each channel that delivered and the route that supplied it, and sends to that route’s destination as it is now (or the channel’s own destination when no route supplied it). A route deleted or disabled since the fire gets nothing, with or without a restart between the fire and the clear: nothing is sent to that channel, and the history row says so. A mute rule stops a resolution when it matches the alert’s fire, including a rule scoped to a database, wait type, job or query. Each resolution carries the server, the alert, when it fired and cleared, how long it lasted, the last value its fire sent and the fire’s triage link. In alert history, the resolution row records what its send did: get_alert_history reports resolution_sent_at (UTC, set when a channel delivered), resolution_channels and resolution_outcome (sent, failed: <channels>, muted or route deleted or disabled: <channels>), null on every other row; the Viewer’s Alert History grid has a Resolution column and Alert Details shows the three fields; the web Alerts page shows them in the row’s detail. A route can override the channel setting for its own destinations: in the route dialog, Send resolutions is Use channel setting (the default), Always or Never. PagerDuty is not one of the four: its incidents close through Auto-resolve the incident when the alert’s condition clears. Over MCP, get_alert_settings reports the four settings as delivery.send_resolutions, update_alert_settings changes them, and get_notification_routes reports each route’s send_resolutions (null means the channel decides). A bring-your-own store needs tools/provision-roles.sql run again after the service upgrades the store to V187, so the viewer and mcp roles can read the new columns.