This page describes Darling v3.13.0. The source of this page is on GitHub at that version.
Audience: operators.
This page covers how an alert reaches your own agent, the practice requests, the export of the judged cases, and the logins for systems that send alerts. It moved here from the Darling operator guide.
Hand an alert to your own agent (#5632)
Darling ships no agent. It can hand an alert to one you run yourself. The route sets how far the agent can go, from reading and reporting to changing alert settings and monitored servers. A gate records every change. The guide Agent dispatch covers each step. In short:
- Turn it on. Set
mcp.enabledandmcp.allowInvestigationstotrueindarling.json, and give MCP a bearer token (network.encryptedTokenornetwork.token). Keepmcp.allowWritesoff. With both on, the write tools win and the investigation tools stay off.--validate-configand the start-up log warn about a setup that cannot work. The warning never changes the exit code or stops the start. - Pick the alerts. In the Viewer’s route editor, set Agent investigation on a route.
Investigate(Level 1) reads and reports.Act(Level 2) also changes alert settings on the request’s server.Act on servers(Level 3) also runs server actions. At Level 3 the Action approval box of the route decides whether a person must approve each server action first.Requiredis the default for a new route. See Level 3 server actions: approval.Offstops it for that route, andInheritasks the next broader route. The route also sets a limit per hour (20 by default) and per day (100). It sets a lease (15 minutes) and an expiry (4 hours) too. A muted alert, a recovery, a scheduled report and a digest never open a request. - The agent’s loop.
wait_for_investigation_requests(orlist_investigation_requests) finds a request.claim_investigation_requesttakes it and returns the brief.get_investigation_evidencereads the evidence that Darling froze.submit_investigation_reportends the work. At Level 2 and Level 3, the agent changes things first and reports last, because writes refuse after the report. A report can ask for a later follow-up request withfollow_up_at. - Read the results. The Viewer’s Manage Investigations… window and the web alert page show each request, its brief, its report and its journal. In the Viewer, a person can close a request and approve or reject a note. A person can also write a note and set the environment profile. Proposed server actions wait on the web dashboard’s Proposals page and in the Viewer’s Proposals grid. The Agent changes list shows every change an agent made, and each one has an Undo (see “Agent Changes: List, Undo and the Switch”).
mcp.dataDetail masks the brief and every tool answer at the level you set. AGENTS.example.md is a starting point for your agent’s instructions.
At Level 1 an agent reads and reports. At Level 2 it can change alert settings on the request’s server only: it can mute one metric, write or change a custom alert rule, set a per-server threshold override and merge deadlock signatures (see “Level 2: agents change mute rules and custom alert rules”). An agent token can also create, change, enable, disable and delete a mute or digest rule with no claim if the rule names a metric, expires within 7 days of its creation (an agent can shorten that but only an operator can extend it) and the call gives a reason. A digest may name one server or all servers; a mute names one server and may not name one of the service’s own alerts, as under a claim. An operator can turn this off with mcp.allowAgentMuteRules set to false in darling.json (default true), which leaves the claim-based tools as they are. At Level 3 it can also act on the monitored server. It can force or unforce a Query Store plan, evict a cached plan and recompile a module. It can also create or drop a plan guide and kill a session. On a route whose approval is required, the agent only proposes the action, and a person approves it (how approval works). A server action needs the server’s own action switch and a remediation login (see the remediationUsername key and “Ending a PostgreSQL session”). An agent turns an alert off only with a mute or a digest rule, holds the service’s own alerts back only with a digest through the no-claim path with its 7-day bound, and gives a reason for every change. A note that an agent proposes waits for a person to approve it before it reaches a later brief.
Tell Darling whether a report was right (#5913). Open a reported request and say whether its report was right, partly right or wrong, what it should have said (real, false_positive or needs_person), the step the agent missed, and a comment. Do it in the viewer’s Investigations window, on the Report Feedback tab, or on the web Admin page’s Report Feedback tab. Each person has one feedback per report; saving again replaces it, and the author is the login you are signed in as. Feedback is recorded only under the admin login: the viewer needs postgres.connectAs set to "admin", and on a store where the web connects as the web role, which cannot write feedback (managed and compose stores) the web tab lists feedback with no form, says so, and feedback is given in the viewer instead (the web asks the store whether its login may write, so an admin sign-in on such a store gets no form either). Feedback feeds two things. get_investigation_reports returns each report’s feedback next to it, and, for a time window, a count per alert family of how many investigations ended as false positives, once as the agent reported them and once after people’s corrections (a correction can raise or lower the count). The settled verdict of a request is the corrected verdict of the newest feedback that gave one; a later feedback that gives no corrected verdict does not displace an earlier one that did. The same counts are at the top of both Report Feedback tabs, each stating its window: the viewer’s tab defaults to the requests opened in the last 30 days (a 7, 30 or 90 day picker), the web tab to the last 7 days; both list the newest 200 reports, and the counts cover every reported request in the window, not only the listed ones.
Practice requests: how an agent does on cases you already judged (#5918)
Before you trust an agent with Level 2, watch it work cases you have already judged. A practice request replays a reported case that has feedback (see “Tell Darling whether a report was right” above) and scores the agent’s report against that feedback.
- Start one in the Viewer. In Manage Investigations… open the Practice tab, pick a judged case (a reported request somebody gave feedback on), and press Start Practice Request. The viewer needs an admin seat; a read-only seat can read the scores but not start a request. A case with no report, no feedback, or that is itself a practice request cannot be replayed, and a case with a practice request still open is not replayed twice.
- The agent claims it like any request. The practice request copies the case’s route, server, family, alert count and alerts, and is stamped Level 1. The brief it gets says it is practice and holds none of the case’s answers: no earlier report, no prior link, no note that the case’s report proposed, and no verdict, corrected verdict, missed step or comment. The agent cannot read the replayed case while its practice is live: under the investigations arm (
mcp.allowInvestigations, writes off),get_investigation_reportsleaves out the case’s report, feedback and journal for as long as a practice request replaying it reads open or claimed (not once its report is submitted, or it has expired); the case’s request row and every other report still show. The same holds for every other brief: while that practice is live, no other brief’searlier_reportsorprior_linkscarries the case’s report, and a practice request’s own report and links are never history for any brief. The writes arm (an operator’s token) and the web Admin page read everything.list_investigation_requestsand the Viewer’s Requests tab show which requests are practice: a practice row carriespractice_of, the id of the case it replays, and every other row has no such field. The family counts in a window read still count the case, as one among many; they never count a practice request. The score stays advisory. - A practice request is not real traffic. Every Level 2 action on it is refused with
practice, a live alert never joins it, it does not count against the route’s hourly or daily limit, and the Report Feedback list and the per-family counts leave it out; its report shows only in the Practice scores. An approved note that the replayed case proposed still reaches other briefs, as any approved note does. - The score. The agent’s report verdict is compared with the answer the feedback settles on. The newest corrected verdict, if anyone gave one, is the answer. Otherwise
rightmakes the original report’s verdict the answer,partlymakes it the answer at half credit, andwrongwith no corrected verdict names no answer, so that case is shown as “No answer” and is not scored. The score is points over possible credit per agent, counting each agent’s newest report on each case. Cases whose feedback names a missed step are counted so you can read the practice report against that step yourself; Darling does not judge the step. - Where scores show. The Viewer’s Practice tab, and the web dashboard’s Admin page, Practice Scores tab (list-only). The score is advisory: it turns no route on or off.
Provisioning: the admin role holds INSERT on config.investigation_requests (the columns a request is opened with) and config.investigation_request_alerts, so the Viewer can open a practice request. Run the provisioning script again on a bring-your-own store; managed and compose stores re-apply it on start.
Export the judged cases to test your own agent (#6236)
Every case somebody gave feedback on (see “Tell Darling whether a report was right” above) is a labelled test case: you know what the alert was, what the agent was given, what it concluded and whether a person agreed. The export writes them all out so you can replay them against your own agent.
PerformanceMonitor.Darling.Service.exe --export-investigations cases.jsonl [--since 2026-09-01] [--until 2026-10-01] [--server <name>] [--verdict wrong] [--force] [--config <path>]
- One case per line, JSON lines. Each line is one object:
request_id,alert(the server, the family, the first metric, when and how often it fired),brief(the alert and the evidence the agent was given),report(what it wrote: verdict, root cause, summary, evidence by basis) andfeedback(every verdict, corrected verdict, missed step and comment people recorded, with the author). - Filters.
--sinceand--untilare dates (a date alone is its midnight UTC) and select by when the request was opened;--sincemust be before--until.--servertakes one registered server’s storage name or display name exactly (case does not matter, a part of a name matches nothing, and a name two registrations share is refused with both listed); a server that was disabled can still be exported.--verdictkeeps the cases a person markedright,partlyorwrong. Without filters you get every case with feedback, newest first, up to 2000; the verb says so when the filter held more. A practice request is not a case and is left out, and so is a case a live practice request replays (the verb says how many were held back). While the practice is live that case is also taken out of every other case’s earlier reports and prior links, in the exported brief and in the stored briefget_investigation_reportsserves an investigating agent; when the practice ends it is back in all of them. - The file. It is written beside the target as a hidden partial file, then moved into place. The partial file is readable by you only on Linux and macOS, and on Windows by you, the administrators and the service’s operator group (the same people who can read
darling.json). Ctrl+C or a failure deletes it, and if it cannot be deleted the verb prints its path. A folder, a link or an existing file is refused (an existing file is replaced only with--force, and one that appears while the store is being read is never replaced without it). The verb prints the full path it wrote. - Masked as the reports are, on both paths. Query text and other collected text are masked at your
mcp.dataDetaillevel, the wayget_investigation_reportsmasks them (amaskLiteralsormetadataOnlyinstall exports masked text, with the same login numbering), and a password statement in a report, a comment or the brief is withheld at every level. The command line and the web download run the same code, so the same cases at the same level give the same bytes. The command line readsmcp.dataDetailfrom the file when it runs (the verb prints the level it used); the web download uses the level the service read when it started. - From the web dashboard.
GET /api/investigation-export(the same filters as query parameters:since,until,server,verdict) downloads the same lines asinvestigation-cases.jsonl. Only a signed-in dashboard account with the admin role may download: a viewer account gets a 403, and so does a dashboard with no sign-in configured. The store login must also be able to record feedback, theadminrole (postgres.webConnectionStringon a bring-your-own store); thewebrole gets a 403. The route counts as a heavy read from its first call, and only one export runs at a time: a second answers busy (429 with a retry-after) until the first has been sent. The read slot is held while the store is read and released before the file is sent. The cases are written one at a time to a file in the service’s own data folder (anexportsfolder beside the store’s data folder), which is deleted when the send ends and removed by the system if the service stops in the middle. No MCP tool serves the export.
How to use it. After you change your agent’s prompt or model, replay the cases: give your agent each line’s alert and brief the way a claimed request would, collect its new verdict, and compare it with the recorded feedback. A person’s corrected verdict, when there is one, is the answer; otherwise right means the original report.verdict was correct, and wrong means it was not. Count where the new agent agrees, and read the cases where it now disagrees. For a live trial inside Darling use a practice request (above), which scores the agent for you.
Make a login for a system that sends alerts (--add-alert-source and friends)
A system outside Darling can hand Darling an alert to deliver through your email, webhook and PagerDuty channels. Each such system is a source: a registry entry and a database login of its own. The login can do one thing, call config.submit_external_alert, and Darling takes the source name from the login that made the call, so one source cannot send under another’s name. These four verbs make and manage the logins. They run as the store owner, the way --add-server does, and each takes an optional --config <path>.
PerformanceMonitor.Darling.Service.exe --add-alert-source orders
PerformanceMonitor.Darling.Service.exe --rotate-alert-source orders
PerformanceMonitor.Darling.Service.exe --list-alert-sources
PerformanceMonitor.Darling.Service.exe --remove-alert-source orders
-
--add-alert-source <name>makes the logindarling_src_<name>, registers it and gives it the grants, then prints its connection string. The connection string is shown once. Darling keeps no copy of the password and the password is never sent to the server as text: the server receives only a SCRAM-SHA-256 verifier. Save the string where the sending system can read it. A name is 2 to 31 characters: a lower-case letter, then lower-case letters, digits or underscores. -
--rotate-alert-source <name>sets a new password and prints the new connection string once. The old password stops working at once, for new connections, and the verb then ends the sessions the login already holds and says how many. Use it when a string is lost or leaves the people who should have it. -
--list-alert-sourcesprints each source’s name, state, when it was created, when it last sent an alert and how many alerts of its are waiting for delivery. It shows no password, because none is stored. The last-sent time and the waiting count come from the queue rows the store still keeps. -
--remove-alert-source <name>removes the source, drops its login and ends the sessions the login still holds. The alerts it has queued but Darling has not yet delivered are removed with it, and the verb says how many. Alerts it already sent stay in the alert history.
The source then sends an alert with one statement, for example SELECT config.submit_external_alert('fire', 'disk-full', 'warning', 'Disk almost full', '93 percent used', 'web01', 93.5);. The first argument is fire or clear, the second is a key of your choosing that pairs a clear with its fire, and the third is warning or critical.
The host in the printed string is the address this service uses for its store. If the store’s owner cannot create logins (a bring-your-own store whose owner lacks CREATEROLE), the verb changes the registry and prints the statements a database administrator runs to make, re-key or drop the login. Those statements end with the login’s membership in darling_alert_sources (below); tools/provision-roles.sql makes that group.
Alert sources on another machine
Every source login is a member of darling_alert_sources, a NOLOGIN group role that holds no privilege on anything. The service creates it, marks it, and on every start makes the registered source logins its only members (on PostgreSQL 16 and later the membership is granted WITH INHERIT FALSE, SET FALSE). A role of that name that Darling did not create stops provisioning; it is never taken over.
For a managed store:
- Make the store reachable from the network (
postgres.network.listenandallowFrom, see Remote viewer). - Add
sourcestopostgres.network.role, keeping the roles it already lists, for example"role": "viewer,sources". - Restart the service. The managed pg_hba block gains
hostssl darling +darling_alert_sources <allowFrom> scram-sha-256, which admits, from that range over TLS, each role that is a member ofdarling_alert_sourcesdirectly or through another role. The service keeps those members to the registered source logins, and writes the line only while they are exactly that: if a member remains that it could not revoke, it leaves the line out (the block’s other lines stay) and logs a warning naming the role. -
--add-alert-sourceand--rotate-alert-sourceprint two connection strings: one for a source on this machine, and one for a source on another machine, built the way the remote viewer’s is: the store’s network address,SSL Mode=VerifyFullandRoot Certificatenaming the store’sserver.crt. Copy that file to the source machine and pointRoot Certificateat the copy. Both strings are shown once. Whensourcesis not listed yet, the second string is still printed, with a line saying it works oncesourcesis listed.
For a bring-your-own store the second string uses the store address from the string the verb connected with, SSL Mode=VerifyFull, and a Root Certificate placeholder: replace it with the path of the CA certificate that signed the store server’s certificate, and set Host to a name or address that certificate carries.
When the store owner may not end a login’s sessions, or may not turn utility statement tracking off (pg_stat_statements.track_utility, which the verbs turn off for their own transaction so the role statement is not recorded), the verb says so and prints the statement a database administrator runs.
A source added later is admitted as soon as --add-alert-source returns; there is nothing to restart. For a bring-your-own store, add the same line to your own pg_hba.conf (with your database name and range) and reload it. On PostgreSQL 16 and later, a role that is not a superuser and creates the group or a source login (the store owner running tools/provision-roles.sql or the add verb, or an administrator running the printed statements) becomes a member of it and cannot revoke that itself; the verbs and the script name every such role, with a hostssl <database> <role> <range> reject line to put ABOVE the group line.
A source login can read what every PostgreSQL login can: the system catalogs and statistics views, such as pg_roles, pg_stat_activity and function definitions in pg_proc. In the store it can call config.submit_external_alert and nothing else: no table, view or other function of Darling’s, and not pg_stat_statements (the service and the script take that view from PUBLIC). TimescaleDB’s own schemas are not restricted: timescaledb_information, _timescaledb_catalog and _timescaledb_config are readable by every login, and show the store’s hypertables, chunks and background job settings (names, intervals and policy configuration, not collected data).