This page describes Darling v3.13.0. The source of this page is on GitHub at that version.
Audience: operators.
This page covers the store’s schemas and roles, the list of agent changes with its undo and its switch, and what an agent can change at Level 2. It moved here from the Darling operator guide.
Security & Least-Privilege Roles
The store is split into two schemas so that no consumer connects with more privilege than it needs:
-
collect— the collector hypertables (one per collector) plus the service-written, user-read metadata (servers,collection_log,analysis_findings, thev_*views). Read-only to everyone but the service. -
config— exactly the tables a human operator changes through the Viewer or MCP:config_mute_rules,config_alert_log(alert dismissals),config_edge_trigger_watermarks, andanalysis_muted.
Table names are unchanged — only their schema moved — and the shared SQL keeps using the bare, unqualified names, resolved through search_path = collect, config, public (set as the database default and carried on the managed connection strings). This is deliberate: Darling’s SQL is byte-identical to Lite’s DuckDB SQL, and re-qualifying it would fork that twin.
The roles. The service still owns the store as the darling superuser (it does the DDL — migrations, hypertable conversion, retention). On top of that, managed mode provisions four least-privilege login roles, and so does the service on the compose distribution’s own store; any other bring-your-own store gets the same four from a shipped script (see below):
| Role | Privileges | Used by |
|---|---|---|
darling |
superuser / owner | the service (collection, migration, provisioning) |
admin |
SELECT on both schemas — including the secret columns, which the Settings window reads — plus INSERT/UPDATE/DELETE on config only. No statement timeout |
the Viewer, by default (connectAs: "admin") |
viewer |
SELECT on all of collect, and on config minus the secret columns of config_monitored_servers / config_command / config_notification (carved fail-closed, below). No writes. Runs under statement_timeout = 15s |
a locked-down Viewer (connectAs: "viewer") and remote Viewer seats |
web |
viewer‘s exact read surface (the same secret-column carve) + the web dashboard’s narrow single-table writes: INSERT/UPDATE/DELETE on config.custom_views (the composer’s saved views), on config.custom_alert_rules (the user-authored alert rules), on config.database_state_expected (the per-database override editor) and on config.config_mute_rules (the dedicated mute-rule endpoints, #3450) — plus UPDATE on exactly the two config_service reload-beacon columns the mute table’s bump trigger writes as the caller. Runs under statement_timeout = 15s |
the web dashboard (managed and compose stores: its own login; bring-your-own: when postgres.webConnectionString names it). Never a seat login, and it never gets a network line |
mcp |
viewer‘s exact read surface + INSERT on collect.analysis_findings + INSERT/DELETE on config.analysis_muted (the mute and unmute tools) + INSERT/UPDATE/DELETE on config.custom_views (the custom-view tools) + INSERT/UPDATE/DELETE on config.database_state_expected (set_database_state_expected, #6397) + UPDATE of the three close columns on config.investigation_requests and INSERT of the five decided-note columns and UPDATE of the four decision columns on config.investigation_notes (the four investigation admin tools, #6400) + INSERT of the practice request’s columns on config.investigation_requests and of the copied alert columns on config.investigation_request_alerts, INSERT/UPDATE of the feedback columns on config.investigation_feedback and INSERT/UPDATE of the profile columns plus DELETE on config.agent_environment_profiles (the three tools that start a practice request, record feedback and save a profile, #6400) + the alert-tuning writes (INSERT/UPDATE/DELETE on config.config_mute_rules, UPDATE on config.config_alert_settings, UPDATE on the single non-secret email_cooldown_minutes column of config.config_notification, and the config_service reload-beacon columns) + the server-onboarding writes (INSERT/UPDATE/DELETE on config.config_monitored_servers — the credential column stays SELECT-carved, so it can WRITE a password blob but never READ one back) |
the store identity the MCP host connects as — loopback and network alike — on a managed store (DarlingMcpHostService builds its pool from the stored mcp-role credential, never from the owner’s) and on the compose store; on any other bring-your-own store when postgres.mcpConnectionString names it |
admin cannot DROP, alter schema, touch collect data, or create objects — it can only do what the Viewer’s mute-rule / alert-dismiss surfaces need. The mcp role is narrower still: it reads exactly what viewer reads (the secret config columns are carved out identically) and its writes are a small, enumerated set — the two analysis-table INSERTs (analyze_server + mute_analysis_finding), the single-table config.custom_views CRUD (the custom-view tools), the alert-tuning writes (config.config_mute_rules CRUD + a single-row config.config_alert_settings UPDATE + a single-COLUMN config.config_notification UPDATE for the delivery cooldown, plus the two config_service beacon columns so a settings write’s self-bump trigger can fire), and the server-onboarding writes (config.config_monitored_servers CRUD for add_servers / remove_server — its config_monitored_servers write fires the SAME config_service beacon trigger, already covered by that column grant) — so a token-holder on the network MCP endpoint can never reach the config-table service-credential pivot, the secret columns, or a service flag like paused. Even on config_monitored_servers, which it may write, the encrypted_password column stays in the fail-closed secret carve, so mcp can WRITE a credential blob (onboarding) but can never READ one back. config_notification is the sharpest case of the same shape: it holds the SMTP password blob, the Teams/Slack/generic webhook URLs and the PagerDuty routing key, so the write there is COLUMN-level on the one non-secret cooldown column — mcp cannot write even a non-secret sibling like smtp_host, let alone a credential. ALTER DEFAULT PRIVILEGES means new collector tables auto-inherit SELECT for admin/viewer/web, so the model never drifts as collectors are added (every mcp write is an explicit single-table/single-column grant, deliberately not schema-wide).
Managed mode provisions all of this automatically on every start (idempotent and self-healing), generating a per-role DPAPI-LocalMachine credential — pg-admin-credential.dpapi, pg-viewer-credential.dpapi, pg-web-credential.dpapi, and pg-mcp-credential.dpapi beside the data directory, same posture as the owner’s pg-credential.dpapi. The passwords never travel in a statement: provisioning sends each role’s SCRAM-SHA-256 verifier, which PostgreSQL stores as-is (what psql’s \password does), and only for a role whose stored verifier no longer matches its credential file, so a steady-state start sends none (#3910). Every start also switches off SUPERUSER, CREATEROLE, CREATEDB, REPLICATION and BYPASSRLS on the four roles and revokes any role membership someone granted one of them since (#3914): the batch grants none, and either would outrank the grants above (a pg_read_all_data membership reads the carved secret columns). Nothing to configure beyond connectAs.
The compose store gets the same roles from the same batch (#3914), because it is the service’s own: the service recognizes it by logging in as the store’s bootstrap superuser (the role initdb created, POSTGRES_USER in the compose file), inside a container. The passwords are generated the same way and kept as plain owner-only files (pg-admin-credential, pg-viewer-credential, pg-web-credential, pg-mcp-credential, the file: secret shape) in the darling-credentials volume at /var/lib/darling/credentials — the container boundary is the isolation there, as it is for secrets/. Three differences, each deliberate: the roles carry the comment darling-compose rather than darling-managed, so the service never takes over a role provision-roles.sql made (it re-keys the roles it owns from its credential files every start, and would otherwise reset an operator’s password); the owner and database are whatever the store’s login and database are called; and the database-level REVOKE ALL … FROM PUBLIC is left out, because people read the compose store directly with roles of their own and that statement would take their CONNECT away. When the store is not recognized (a login that is not the bootstrap superuser, or a cluster that also holds a database besides the store’s own and postgres, which makes it an operator’s own cluster), or a same-named role is not the service’s, nothing is created. On such a start, and on one whose provisioning fails or never runs because the collector stopped first, the web dashboard and MCP keep connecting as web and mcp with the credentials an earlier start wrote, if the files pass the trust checks; a role login the store refuses keeps that surface down rather than falling back. Only a surface with no usable credential connects as the owner. Each surface’s startup warning says which of these it is and why. The trust checks: every start sets the credentials directory back to owner-only (0700) before reading it, and a directory that is a symbolic link, still reachable by other users afterwards, or was writable by them before has none of its files read (every role gets a new password that start). Each file must be a regular file with no group or other permissions. The owner of a role credential file is not checked; the named volume the Dockerfile seeds as owned by user 1654, mode 0700, is what keeps anyone else from creating one, so a bind mount another host user can write to is outside what the checks can see. The owner of the credentials directory is checked (see Container user), and so is the owner of the password key file, by statx or stat.
Container user
The image runs as user 1654 ($APP_UID of the .NET base image), not as root. /var/lib/darling and the file log are owned by it, and the darling-credentials volume is seeded owned by it, mode 0700. setup.sh writes darling.json and secrets/* as 0644 inside a 0700 secrets/, so the service can read them. Never put a secret inline in darling.json. The one exception to the 0644 files is a password key kept as a compose secret (passwordKey: file:/run/secrets/password_key): on the host it is owned by 1654 with mode 0400 or 0600, and it is the only file the deploying user chowns (see the password key below, and secrets/README.md).
Upgrading an install that ran as root. Its credentials volume is owned by root. Run these once in the compose folder, after docker compose pull and before docker compose up -d starts the new image:
chmod 700 secrets && chmod 644 darling.json && find secrets -type f ! -name password_key -exec chmod 644 {} +
docker compose run --rm --no-deps --user 0 --entrypoint chown darling -R 1654:1654 /var/lib/darling
The first line keeps secrets/ closed to other users on the host while its files become readable by the container’s user. It leaves a file named password_key (the password key kept as a compose secret) alone, so the key stays owned by 1654 with mode 0400 or 0600 (see secrets/README.md). If the new image started first, what it logs depends on the line that was skipped. Without the chmod, it cannot read the first secret it loads (3.12’s setup.sh wrote secrets/* as 0600 files of the host user), so it cannot load darling.json and collection does not start: the log says the secret file could not be read, names user 1654, and gives both lines. With the chmod run but not the chown, it finds a credentials directory owned by another user and stops collection: the container is unhealthy, the web dashboard and MCP stay down, nothing in the directory is read or changed, and the log line gives the chown. Either way, run the two lines, then docker compose restart darling. docker compose up -d leaves a running container as it is, so collection would stay stopped.
Stopgap. user: "0:0" on the darling service (a commented line in docker-compose.yml) keeps the old behaviour, but only before the chown step above has run. After it, a root service refuses the 1654-owned key file and names both users.
Rollback. Run the same command with -R 0:0:
docker compose run --rm --no-deps --user 0 --entrypoint chown darling -R 0:0 /var/lib/darling
Kubernetes. The Helm chart sets all of this for you. To write the manifests yourself, mount a volume at /var/lib/darling and run the pod as 1654:
spec:
replicas: 1
strategy:
type: Recreate
template:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1654
runAsGroup: 1654
fsGroup: 1654
fsGroupChangePolicy: OnRootMismatch
containers:
- name: darling
volumeMounts:
- name: darling-state
mountPath: /var/lib/darling
-
A new volume needs nothing more, on a volume type kubelet applies
fsGroupto. (On one that ignoresfsGroup, a root-owned volume root needs the one-off Job below too, or the service as 1654 cannot createcredentials/.) On the first mount kubelet gives the volume root group 1654, group rwx and setgid. The service makescredentials/itself, owned by 1654, mode 0700, andOnRootMismatchleaves the volume alone on every later mount, because its root already matches. -
An install that ran as root needs a one-off Job (or an initContainer) that runs as root with the volume mounted at
/var/lib/darling, before the new pod starts. With a ReadWriteOnce volume, scale the Deployment to 0 first so the Job can attach the volume, run the Job, then scale the Deployment back:chown -R 1654:1654 /var/lib/darling && chmod 2770 /var/lib/darling && chmod 700 /var/lib/darling/credentials && chmod 600 /var/lib/darling/credentials/*The
chmod 2770gives the volume root whatOnRootMismatchchecks for (group 1654, group rwx, setgid), so kubelet skips its recursive walk andcredentials/stays 0700.fsGroupalone does not fix such a volume: its root does not match, so the first mount walks it,credentials/stays owned by root, and the new image stops collection with thechownin its log. The Job still fixes it after that, and the keys are kept. -
A volume mounted at
credentials/(the compose shape) has two choices. Keep that mount and dropfsGroup, with a Job that mounts the volume there and runschown -R 1654:1654 /var/lib/darling/credentials && chmod 700 /var/lib/darling/credentials && chmod 600 /var/lib/darling/credentials/*. Or move its files into acredentials/subdirectory in the Job above, mounted at/var/lib/darling, by runningcd /var/lib/darling && mkdir .moving && mv -- * .moving/ && mv .moving credentialsbefore the commands above, and then mount the volume at/var/lib/darling. -
Never combine
fsGroupwith a mount atcredentials/, and never setfsGroupChangePolicy: Always. Kubelet appliesfsGroupwhen it mounts the volume, so every time the pod starts (not on a container restart inside the same pod), on a volume type it appliesfsGroupto, it would opencredentials/to the group, and the #4004 rule would discard the keys each time.
Credential file protection. DPAPI LocalMachine scope is deliberate (the service writes the credential, a different interactive user’s Viewer reads it), which means the machine-bound blob is decryptable by anything that can read the file. So the credential files are locked down with an NTFS ACL that strips the inherited world-read %ProgramData% would give them:
| File(s) | Readable by |
|---|---|
pg-credential.dpapi (superuser), pg-web-credential.dpapi (the web dashboard’s role), pg-mcp-credential.dpapi (the network MCP role) + the transient init pwfile |
SYSTEM, Administrators, the service account. Not the operator group |
pg-admin-credential.dpapi, pg-viewer-credential.dpapi |
the above + members of the local group PerformanceMonitor Darling Operators (the Viewer and the CLI, run without elevation) |
log-hash-key.dpapi (the store’s log-hash key, #4004) |
SYSTEM, Administrators, the service account. Not the operator group |
pg-web-credential.dpapi and pg-mcp-credential.dpapi sit with the superuser (non-interactive) rather than with the Viewer’s credentials because only the in-service web and MCP hosts read them — never an interactive Viewer.
The log-hash key (#4004). pg_log_events stores two identities per event: raw_line_hash, which every read dedupes on, and statement_fingerprint, which groups one statement shape. Both are HMAC-SHA-256 under a 32-byte key the service generates and keeps outside the store, so a reader of the store cannot test guesses at the literals a log line hid (an unkeyed hash let #3996’s review recover a four-digit PIN in 6 ms). Neither hash is returned by any tool or viewer. The key is loaded once at service start and lives beside each distribution’s other secrets: log-hash-key.dpapi next to pg-credential.dpapi on a managed store (DPAPI, the ACL above); log-hash-key in the darling-credentials volume on compose (0600, under the same directory and file checks as the role passwords); and darling-keys/log-hash-key (.dpapi on Windows) beside darling.json on bring-your-own. It is generated when no file exists. The service does not replace an existing key, because a new one gives every stored log event a new identity: a key file it cannot trust or read makes it refuse to collect log events, with the file and the reason in the service log, until you fix the file’s access or delete it (the way to rotate). The one exception is a key directory other users could write to, such as a bind-mounted credentials directory left at 0777: any file in it could have been planted, so the moment the service finds it open (whichever start, and whichever part of the service looks first) it sets it owner-only, removes the key and the role passwords there, and logs an error; a new key and new passwords are generated, and the first pg_log_events run after that says so on its Collection Log row. A directory with something in it at one of those names is left alone (the service never deletes someone else’s tree, and nothing reads a directory as a credential), but everything else is still removed and the directory stays owner-only; that start uses nothing in the directory, and its error names each path to remove before you restart. A directory that keeps being opened (a Kubernetes fsGroup re-applied on every mount of credentials/) rotates the key every time the pod starts, so keep it owner-only and mount its parent, as in Container user. Run one service per credentials volume (on Kubernetes, one replica with strategy: Recreate, never a rolling update): the check is not locked across processes, so a second service starting beside the first could read the directory between the first one’s chmod and its removal. Keep it with the store’s backups of its other credentials. Rows stored before #4004 keep their unkeyed hashes until the 30-day retention drops them, so the first cycle after the upgrade stores each entry still in the tail’s overlap window once more, and a statement shape’s fingerprint changes once.
The password key (#5366). A SQL login password saved in the store is sealed to a key pair the service owns. This works on every platform, so you can save a password on Linux and in a container as well as on Windows. The store publishes the public half, and every place that saves a password seals it to that half: the Viewer, the web dashboard, the MCP tools and --add-server. Only the service holds the private half, so only the service can open a saved password. A sealed password opens only for the connection it was saved with. If you change a server’s host, port, database, login or encryption settings, enter its password again.
- Where the key is kept. In the compose deployment it is the
password-keyfile in thedarling-credentialsvolume, with the other service credentials. On a Linux or Windows install where you bring your own PostgreSQL it is in adarling-keysdirectory besidedarling.json. With the managed PostgreSQL on Windows it ispassword-key.dpapi, beside the other credential files. It is never in the store or its volume. - First start and the fingerprint. The first start makes the key. Each start logs a
Password keyline with the key’s fingerprint, as four groups of four letters and digits. The first time a Viewer saves a password to a store it shows the same fingerprint. Check that the two match. If the key later changes, the Viewer stops saving passwords. It asks you to compare the new fingerprint with the service log, then choose Trust the new key. - Backups. Back up the credentials volume (or the
darling-keysdirectory) after the first start and again after every reset. Keep that backup apart from the store’s backups. A store restored without its key cannot open its saved passwords, and the service reports the key as missing. On Windows the key is protected with DPAPI for the machine, so its backup restores only on the same machine. - Resetting the key.
--reset-password-keyretires the key at the next start and makes a new one. Every saved password must then be entered again. If the key may have been copied, reset it. Then change the SQL logins’ passwords. Delete the retired key file (password-key.retired, orpassword-key.dpapi.retiredon Windows) once you no longer need it. - Keeping the key outside the service (#8006). Set
passwordKeyindarling.jsontofile:<path>orenv:<NAME>, for examplefile:/run/secrets/password_keyorenv:DARLING_PASSWORD_KEY. The service then reads the key from that secret, such as a Kubernetes Secret, and not from the credentials directory. Without the setting nothing changes. A literal key inpasswordKeyis a configuration error. Preferfile:. An environment variable’s value shows indocker inspectwhen it is passed with-e, and inkubectl exec … env(a KubernetessecretKeyRefkeeps it out of the pod spec, which shows only the reference). On Linux root and the service’s own user can read it in/proc/<pid>/environfor as long as the process runs. On Windows a service’s variables are kept in the registry, in the service’sEnvironmentvalue or in the machine environment, and every local user can read both. The service removes the variable from its own environment at start, so the processes it starts do not inherit it, but that does not remove it from/proc/<pid>/environ.env:is still supported. Afile:path inside the credentials directory is refused: mount the secret outside it. A compose secret is a bind mount of a file on the host, so the container sees the host file’s owner and mode. Give that file to user 1654, the container’s user, with mode 0400 or 0600 (chmod 600 secrets/password_key && sudo chown 1654:1654 secrets/password_key). It is the one file insecrets/that is not 0644 and the only one youchown(#8141, seesecrets/README.md). A key file that another user owns and the service cannot read is refused with both users named and thechownthat fixes it. Afile:key does not get every check the default key file gets. The service did not write the file, so it does not own, harden or move it: its owner and mode are yours to set. The key is refused when the path, every link followed, is not a regular file (a directory, a device, a FIFO, a socket, or a pipe reached through/dev/stdinor/proc/self/fd), when the file is larger than 8192 bytes, on Linux when its group or other users can write it, and on Windows when Users, Authenticated Users, Everyone or INTERACTIVE can write it, or when its folder is owned by, or can be changed by, an account other than SYSTEM, Administrators or the service’s account. An account can change a folder when it may write to it, delete it or anything in it, or change its permissions or owner. That is the rule the default key file’s folder follows. It applies to the folder the path names, where a junction is judged by its own permissions, and to the folder the file really is in. On Windows a path with more links than the check follows (40, where Windows itself follows up to 63) is refused, because the folder the file really is in is then unknown. Other checks only give a warning that names the file and the check, at each start and in--validate-config, and the key still loads. On Linux they are: the file is owned by a user that is neither root nor the user the service runs as, it has more than one name (a hard link), other users can read it, or it is in a directory that its group or other users can write to and that has no sticky bit. A file owned by root or by the service’s user gives no owner warning: root can replace any file anyway, so a mounted Kubernetes or Swarm secret, which root owns, does not warn about its owner. On Windows they are: Users, Authenticated Users, Everyone or INTERACTIVE can read the file, or it is owned by an account other than SYSTEM, Administrators or the service’s account, which can change who may write it (give it to Administrators withicacls <file> /setowner *S-1-5-32-544). Compared with the default key file, afile:key is not checked for: read access by the file’s group, which is allowed and gives no message; any directory other than the one the path names and the one the file really is in, so the directories above them are never looked at; a symbolic link, which is followed (a Kubernetes mount needs that); or a file swapped in between the check and the read. On Windows the hard-link check does not run, and the checks on the file’s own ACL look only for Users, Authenticated Users, Everyone and INTERACTIVE, not for other accounts, and pass when the service cannot read the file’s ACL. The folder check counts every other account that can change the folder, and refuses a folder whose ACL the service cannot read. On Windows a folder made directly under a drive root, such asC:\keys, inherits write access for Authenticated Users from the root and passes it on to the files in it, so a key there is refused until the folder is locked down. From an elevated prompt, give the folder full control for SYSTEM and Administrators and read for the service’s account, with nothing inherited:icacls C:\keys /inheritance:r /grant:r "*S-1-5-18:(OI)(CI)F" "*S-1-5-32-544:(OI)(CI)F" "NT SERVICE\PerformanceMonitor Darling:(OI)(CI)RX"(name the account the service logs on as, if you changed it). Then make the key file take its permissions from the folder,icacls C:\keys\<file> /reset, and make Administrators the owner of both,icacls C:\keys /setowner *S-1-5-32-544 /T, because an owner can always change who may write. Or keep the key in a folder that only SYSTEM, Administrators and the service’s account can change, that one of them owns, and that the service’s account can read. On Windows--validate-configjudges the key file as the registered service’s account, and as the account that runs it when the service is not registered, and prints which one on a line of its own. A Kubernetes Secret mounted with its default mode, 0644, loads with the warning that other users can read it. SetdefaultMode: 0440and the pod’sfsGroupto avoid it. The key is one line of base64 of the PKCS#8 DER key, or a-----BEGIN PRIVATE KEY-----PEM block, and it must be RSA-3072. In an environment variable use base64: a PEM whose line breaks arrived as the two characters\nis refused. A PKCS#1 key (BEGIN RSA PRIVATE KEY), an encrypted key, more than one key and a key of another size are refused too. The refusal namespasswordKeyand the file or variable, never the key. To make a key, run--new-password-key <path>, oropenssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out password-key.pem. Put the file in your secret store, then delete it. The account the service runs as must be able to read the key.--new-password-keymakes the file readable only by the account that runs it: mode 0600 on Linux, and on Windows SYSTEM, Administrators and that account. So run it as the service’s account (for a container,docker run --rm --user <uid>:<gid> …), or put the key in a store that sets the owner and mode itself, such as a Kubernetes Secret withdefaultModeand the pod’sfsGroup. On Windows nobody can run a command as the default service account, so move the key into your secret store and delete the file, or keep it in a folder locked down as above and runicacls <file> /reset, so that it takes the folder’s permissions. At each start the service loads the key and checks it against the key the store publishes. When the store publishes none, the service publishes this one and logsPassword key <fingerprint> published from passwordKey. Every later start logsPassword key <fingerprint> loaded from passwordKey. A different published key gives the statemismatch, and a source that cannot be read givesrefused, each with the reason in the log. Collection keeps running. With the setting the service never makes, writes or retires a key file. To move an existing key on Linux or in a container, copy thepassword-keyfile from the credentials directory into your secret store (cat password-key; it is already one line of base64 PKCS#8), setpasswordKey, and restart. Check that the log shows the same fingerprint loaded from passwordKey, then delete the old file. Until you do, each start logs once that the file is not used. To reset the key, run--reset-password-key. A start with the old key then gives the statereset_pendingand publishes nothing. Put a new key in thepasswordKeysource, then restart the service. - Updating and rolling back. Update every Viewer that saves passwords to the store. An older Viewer does not know the key and saves passwords in the old format, which the service does not use for a new row. Passwords saved on Windows before the upgrade keep working on the same machine. If you roll the service back to an older version, it cannot open the passwords saved since the upgrade. Enter them again. Before you roll back to a version without
passwordKey, put the key back in the credentials directory aspassword-key. Older versions ignore the setting, asdarling.jsonskips unknown keys, and would report the key as missing. On Linux the file must be one line of base64 of the PKCS#8 DER key, not a PEM block, owned by the service’s user with mode 0600. On Windows an older version reads onlypassword-key.dpapi, so keep that file if you may roll back.
Who can read darling.json and the admin and viewer credentials. SYSTEM, Administrators, the service account, and members of the local group PerformanceMonitor Darling Operators. No one else can, including a user who is logged on at the console. Other services, sandboxed or SSRF socket code and scheduled tasks are shut out too, and so is the group from the superuser and MCP credentials.
install-darling.ps1 and upgrade-darling.ps1 create the group, empty, when it is missing. Adding a member is your decision, and the answer to the installer’s question is No unless you type y. At the end of the install, install-darling.ps1 asks Add <account> to 'PerformanceMonitor Darling Operators'? [y/N], naming the account signed in to the session. When that is not the account that ran the script (you elevated with another administrator account), the prompt says so. For a scripted install, -AddOperator DOMAIN\user adds that account without asking. -NoOperatorPrompt never asks. A host that cannot take an answer (input redirected, or PowerShell started with -NonInteractive) counts as No. On a domain controller no group is created and nothing is asked. upgrade-darling.ps1 never adds a member.
After a Yes, the account also gets a direct grant of the same access the group has. The Viewer and the CLI then work in the session that is already open, with no sign-out. The grant is read on darling.json, read on the store’s pg-admin-credential.dpapi and pg-viewer-credential.dpapi, and traverse on the store folder. The service rewrites those permissions at each start, which drops the direct grant. From then on the group covers the account only after the person has signed out and back in once. A reboot does that. A service restart or an upgrade does not, so a person still in the same session then has to sign out and back in.
uninstall-darling.ps1 removes the grant from those four paths. A plain uninstall keeps the group and its members. uninstall-darling.ps1 -PurgeData removes the group too, but only when the install created it, which it judges by the group’s description. A group an administrator made first is left in place. To remove it sooner, run icacls <path> /remove *<the account's SID> on each path. On a fresh install, the service writes the credential files at its first start. The installer waits up to three minutes for them, so the grant can include them.
For everyone else, and after a No, add each person who runs the Viewer or the CLI without elevation. Use Computer Management > Local Users and Groups, or net localgroup "PerformanceMonitor Darling Operators" DOMAIN\user /add. A person you add that way has to sign out and back in before the membership takes effect.
Members can read darling.json and the store’s admin and viewer credentials, which decrypt every stored password. Add only people who are trusted with those passwords.
A hand install (sc create, no installer) has no group until you create it. Until then the service grants only SYSTEM, Administrators and itself, and it logs a warning that names the group. Create the group, add the people, and restart the service so it grants the group. Anyone who is not a member sees a message that names the group when the Viewer or a CLI verb cannot read the file.
The installer also creates the store folder before the service starts (%ProgramData%\PerformanceMonitorDarling unless postgres.dataDirectory says otherwise). Administrators own it, and it gets the same locked ACL the service applies on every start. An ordinary user can otherwise create and own that folder first.
The service also refuses to trust a credential file that isn’t owned by SYSTEM/Administrators/itself, and regenerates an untrusted role credential.
A read-only (viewer) Viewer degrades gracefully. It probes its own privileges on connect (has_table_privilege), so the mute-rule Add/Edit/Toggle/Delete/Purge buttons and the alert Dismiss / Dismiss All buttons are hidden or disabled, and any write that still slips through returns a clear “read-only connection” message instead of an error.
Bring-your-own PostgreSQL. The schema split runs everywhere (it’s a migration — the service applies it on startup and best-effort sets the database search_path; if your collection login can’t ALTER DATABASE, run that one statement yourself as the owner). The service provisions roles only on a managed store and on the compose store, so for any other BYO store you create them yourself, once, with the shipped script:
psql -X -h <host> -U <owner> -d darling -f Darling/tools/provision-roles.sql
The release zip, the MSI and the Linux tarball carry the same script as provision-roles.sql beside the service exe, so a package install runs it from the install folder. The path above is for a source checkout.
Use psql 10 or later. The script saves and restores your ON_ERROR_STOP around its collision check, and only psql 10 and later read an unset one as off; in psql 9.6 the run stops at its first error, so an owner without superuser gets no grants at all. Edit the four password placeholders (and the database/owner names if yours differ) first. Each placeholder is in two places, a check and its ALTER ROLE line, so use a find-and-replace on the whole file. The script stops with an error, before it sets any password, while any of the four is unchanged. If you re-run it after an upgrade with a routine that replaces CHANGE_ME_ADMIN_PASSWORD, CHANGE_ME_VIEWER_PASSWORD and CHANGE_ME_MCP_PASSWORD only, add the fourth, CHANGE_ME_WEB_PASSWORD, to it. To set the passwords with \password instead, put a long random value in each placeholder for this run, which \password then replaces for each of the four roles. The -X skips your own psqlrc: an \set AUTOCOMMIT off in it would leave steps 3 to 5 (the write grants, the default privileges and the public hardening) uncommitted while psql still exits 0. The roles, the passwords, the read grants, the secret-column carve and the password key table revoke commit at the script’s own COMMIT. An ON_ERROR_STOP in your psqlrc would change where the run stops. It creates the same four login roles, admin, viewer, web and mcp, with exactly the managed grants (a test holds the script’s grants, default privileges and role settings to the managed batch’s). If a role with one of those names already exists without Darling’s darling-managed marker (a comment on the role), the script stops at that check with an error, before it re-keys or grants anything on that role: rename or drop the other role, then run it again.
The guard also refuses a marked role that another login administers. The marker is a public string, so it does not show who made the role. From PostgreSQL 16, a role with CREATEROLE holds ADMIN OPTION on every role it creates. That option lets the holder reset the password of the role and grant the role to other roles. The script refuses a marked role when another login holds ADMIN OPTION on it. That login is any role that is not a superuser, not the database owner and not the role running the script. A plain membership, such as GRANT viewer TO some_analyst, does not count. To continue, run the script as that login, which the check leaves out. If more than one login holds the option, run the script as one of them after the others lose it. Or, as a superuser, revoke that option or drop the role. A login that created the role cannot revoke the option itself, because PostgreSQL records that grant as made by the bootstrap superuser. Then run the script again.
The guard creates the four roles without a password. The ALTER ROLE lines after the guard set the passwords. A run that stops at the guard therefore leaves no password in the server log. The server writes a failed statement to its log with the whole statement text. A run that succeeds also writes all four passwords to the log when log_statement is ddl, mod or all, or when log_min_duration_statement is 0 or more. Step 0 turns both settings off for the session of the script, and the two sampling settings log_min_duration_sample and log_transaction_sample_rate as well. It also sets log_min_error_statement to panic, so that a failed statement is not written to the log with its text, and it turns off debug_print_parse, debug_print_rewritten and debug_print_plan. It can do that for a superuser, and for a role that was granted SET on the setting (PostgreSQL 15 and later). The role and grant statements of the run are then not in the log either. When a role that may not change log_statement or log_min_duration_statement finds either one set so that the passwords are logged, the script prints a WARNING right after step 0 and goes on.
pg_stat_monitor is handled like pg_stat_statements: step 0 sets pg_stat_monitor.pgsm_track_utility = off for the session (any role may), and the guard stops the run if it is still on. pgaudit is different. Its role class writes ALTER ROLE statements to the log, with the statement text while pgaudit.log_statement is on, which is its default. Only a superuser can change a pgaudit setting, so step 0 sets pgaudit.log_statement = off for a superuser (or a role granted SET on it) and for any other role the pgaudit entries of the run keep their statement text. The same holds for its function class, which logs the DO blocks that check the placeholders, so the step 0 warning also names a pgaudit.log setting that covers function. Use a SCRAM-SHA-256 verifier in place of each password literal there.
A store you bring yourself needs one more step, for large objects and session locks. On a managed or compose store the service takes two groups of built-in functions away from every role. It grants the session lock functions back to the logins that use them. The script cannot do this, because the functions belong to a superuser. So a superuser runs the statements below once, connected to the store database. Without the first statement, any login that can connect, a viewer seat included, can store data in the database through large objects. Without the second and third, a session or transaction lock on the migration key, held open, makes the next upgrade wait out its time limit and stop. Replace <owner> with the role your postgres.connectionString uses.
REVOKE EXECUTE ON FUNCTION pg_catalog.lo_create(oid), pg_catalog.lo_creat(integer), pg_catalog.lo_from_bytea(oid, bytea) FROM PUBLIC;
REVOKE EXECUTE ON FUNCTION pg_catalog.pg_advisory_lock(bigint), pg_catalog.pg_advisory_lock(integer, integer),
pg_catalog.pg_try_advisory_lock(bigint), pg_catalog.pg_try_advisory_lock(integer, integer),
pg_catalog.pg_advisory_lock_shared(bigint), pg_catalog.pg_advisory_lock_shared(integer, integer),
pg_catalog.pg_try_advisory_lock_shared(bigint), pg_catalog.pg_try_advisory_lock_shared(integer, integer) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_advisory_lock(bigint), pg_catalog.pg_advisory_lock(integer, integer),
pg_catalog.pg_try_advisory_lock(bigint), pg_catalog.pg_try_advisory_lock(integer, integer) TO <owner>, admin, mcp;
REVOKE EXECUTE ON FUNCTION pg_catalog.pg_advisory_xact_lock(bigint), pg_catalog.pg_advisory_xact_lock(integer, integer),
pg_catalog.pg_try_advisory_xact_lock(bigint), pg_catalog.pg_try_advisory_xact_lock(integer, integer),
pg_catalog.pg_advisory_xact_lock_shared(bigint), pg_catalog.pg_advisory_xact_lock_shared(integer, integer),
pg_catalog.pg_try_advisory_xact_lock_shared(bigint), pg_catalog.pg_try_advisory_xact_lock_shared(integer, integer) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION pg_catalog.pg_advisory_xact_lock(bigint), pg_catalog.pg_advisory_xact_lock(integer, integer),
pg_catalog.pg_try_advisory_xact_lock(bigint), pg_catalog.pg_try_advisory_xact_lock(integer, integer),
pg_catalog.pg_advisory_xact_lock_shared(bigint), pg_catalog.pg_advisory_xact_lock_shared(integer, integer),
pg_catalog.pg_try_advisory_xact_lock_shared(bigint), pg_catalog.pg_try_advisory_xact_lock_shared(integer, integer) TO <owner>, admin, web, mcp;
The service itself uses no large objects. The session lock grants stay on <owner>, admin and mcp because the migration lock and the agent-change undo take session locks. The transaction-level lock functions go to <owner>, admin, web and mcp, the logins whose own code takes one, so a viewer login cannot hold one open.
These statements change the store database only. Each database on the server has its own large object functions and its own TEMP grant. A login that can also connect to postgres or template1 can still store data there, on the same disk. The managed store runs REVOKE CONNECT ON DATABASE postgres, template1 FROM PUBLIC; for that reason. If no other login needs to connect to those two databases, run it on your server too.
Every run also takes back what an adopted role picked up since. It switches off CREATEROLE, CREATEDB, REPLICATION and BYPASSRLS, and it revokes every membership in another role. Managed mode does the same. The attribute statements and the membership statements run as two blocks with ON_ERROR_STOP on: a privilege refusal prints a WARNING and the run goes on, and any other error stops the run, so a lock timeout or a concurrent role change cannot leave a role with an attribute or membership while psql exits 0.
The table grants, the secret-column carve and the revoke on the password key tables (step 2d) run as one transaction. If a statement in that block fails, psql prints the error and none of the block commits. Fix the cause, then run the script again. Step 3g repeats the key table revoke after the write grants of step 3, so the key tables are closed to viewer, web and mcp when the transaction commits and stay closed.
Any other error is printed and the run carries on with the rest of the script. This is psql’s default and is deliberate. Read psql’s output for errors, and run the lines that failed as a superuser.
An owner that is not a superuser can run the script, with these limits:
-
The owner needs
CREATEROLE. From PostgreSQL 16 it also needsADMIN OPTIONon each of the four roles, to set a password or take back an attribute. An owner holds that option only for a role it created itself. For a role that a superuser made, a superuser must first runGRANT <role> TO <owner> WITH ADMIN OPTION. The guard checks this before it sends a password. If the owner cannot change one of the four roles, the run stops at the guard with exit code 3. The message names the roles and theGRANTstatements to run. The same stop happens when one of the roles is a superuser. Before PostgreSQL 16, it also happens when one of the roles hasREPLICATION. Only a superuser can change such a role. The guard also stops the run, before a password is sent, when one of the four roles is a member of a role that would outrank the grants (the owner, a superuser orBYPASSRLSrole,pg_read_all_data,pg_write_all_data,pg_read_server_files,pg_write_server_filesorpg_execute_server_program) and the owner cannot revoke that membership. From PostgreSQL 16 that needsADMIN OPTIONon the role the membership is in, and no role has that on itself. The message names each membership. A superuser is never stopped by these checks. -
The owner cannot set
temp_file_limitor the two slow-statement logging settings, and those lines fail. -
The owner cannot take
pg_stat_statementsfrom PUBLIC when a superuser created the extension (step 1b), because only the owner of the extension or a superuser may. The script and the service then leave the grants as they are, and each prints a warning that names the owner of the extension and the statements to run. Run them as that owner, in the store’s database (with your schema if the extension is not inpublic, and your store owner fordarling; drop the last four lines when there is nopg_stat_statements_infoview):REVOKE SELECT ON public.pg_stat_statements FROM PUBLIC; REVOKE EXECUTE ON FUNCTION public.pg_stat_statements(boolean) FROM PUBLIC; GRANT SELECT ON public.pg_stat_statements TO pg_read_all_stats, darling; GRANT EXECUTE ON FUNCTION public.pg_stat_statements(boolean) TO pg_read_all_stats, darling; REVOKE SELECT ON public.pg_stat_statements_info FROM PUBLIC; REVOKE EXECUTE ON FUNCTION public.pg_stat_statements_info() FROM PUBLIC; GRANT SELECT ON public.pg_stat_statements_info TO pg_read_all_stats, darling; GRANT EXECUTE ON FUNCTION public.pg_stat_statements_info() TO pg_read_all_stats, darling;A superuser who runs the script does this itself, and grants the view to the store owner too, so the service’s statement statistics keep working for an owner that is not a superuser.
-
The owner cannot switch off
SUPERUSER. From PostgreSQL 16, it can switch offREPLICATION,BYPASSRLSandCREATEDBonly if it holds that attribute itself. From 16, it can also revoke a role membership only withADMIN OPTIONon the role that the membership is in. Before PostgreSQL 16, only a superuser can switch offREPLICATIONorBYPASSRLS. The script prints a WARNING for each attribute or membership that it leaves in place. From PostgreSQL 16, the owner also has to be able to act as the role that granted each membership of the four roles (the grantor inpg_auth_members), because a revoke removes only the membership recorded with the grantor it names. The script names that grantor in each revoke. The guard stops the run when the owner cannot act as the grantor of a membership that would outrank the grants, and a run that still finds such a membership after the revokes stops with an error before any grant. -
If
pg_stat_statementsis loaded with utility tracking on, the script’s first statement,SET pg_stat_statements.track_utility = off, fails for this owner. The run then stops at the guard, before it touches a role or a password. The stop matters because utility tracking records the password statements verbatim, where everypg_read_all_statsmember can read them. To continue, runGRANT SET ON PARAMETER pg_stat_statements.track_utility TO <owner>as a superuser (PostgreSQL 15 and later), or turn utility tracking off for the cluster. Then run the script again. -
Two settings put the passwords in the server log even when the run succeeds:
log_statementatddl,modorall, andlog_min_duration_statementat0or more. Step 0 turns those settings off for the session. It cannot do that for this owner unless a superuser ranGRANT SET ON PARAMETER <setting> TO <owner>(PostgreSQL 15 and later); where it could not, the script prints a WARNING after step 0. A failedALTER ROLE ... PASSWORDline, for example one that a password-check module rejects, is also written to the log with its password, unless step 0 could setlog_min_error_statementtopanicfor the session. To keep the passwords out of the log, put a SCRAM-SHA-256 verifier in eachALTER ROLEline in place of the literal. You can also set the passwords with psql’s\password. -
Three more cases write the same lines:
log_transaction_sample_rateabove0,log_min_duration_sampleat0or more together withlog_statement_sample_rateabove0, andpgaudit.logcovering theroleclass (roleorall, and not-role) whilepgaudit.log_statementison. The script prints the same WARNING after step 0 when it sees any of them.
Then point the web dashboard and the MCP server at them with the two settings in the postgres block — the service’s own login stays the owner, for collection:
"postgres": {
"connectionString": "file:/etc/darling/store_owner",
"webConnectionString": "file:/etc/darling/store_web",
"mcpConnectionString": "file:/etc/darling/store_mcp"
}
(literal connection strings work too; the references keep the passwords out of darling.json). Left unset, a surface connects as the owner — the collection login — and none of the role-level protections applies to it: not the secret-column carve, not the narrow write grants, not the statement_timeout backstop. The service logs a warning at startup naming exactly that. A read-only Darling Viewer points its connectionString at the viewer role the same way. viewer is read-only and is the login for remote and locked-down seats. web is the web dashboard’s login: point postgres.webConnectionString at it and never give it to a seat. To move a store that an earlier script set up, set the web password in the script, run it again, and point postgres.webConnectionString at web in the same step. Running the script takes the web dashboard’s writes away from viewer, so a web dashboard still on viewer can read but its writes are refused until you point it at web. The service warns at startup while postgres.webConnectionString names viewer. That script is the authoritative grant list for a BYO store — it is what actually runs, the table above is its summary, and an ALTER DEFAULT PRIVILEGES in it means a store gaining collectors later needs no re-grant for reading them. Re-run it after a schema upgrade to cover new tables and the single-table writes a newer migration adds.
The front door holds every store’s token
A front door (see The front door) keeps the MCP token, the agent-role token and the web token of each store it fronts, in the frontDoor block of darling.json. Treat that file as you treat the postgres connection strings. Use mcpEncryptedToken, mcpEncryptedAgentToken and webEncryptedToken where DPAPI is available, or an env: or file: reference. The service never prints a token in a log line or an error, and no page or MCP tool returns the block.
The roles that count at a store are the roles of the tokens the front door holds, not the roles of the person at the front door. Every web call reaches a store through the front door’s shared-token sign-in, and every MCP call from an operator reaches it with the operator token. A change a person makes through the front door is attributed at the store to that shared sign-in, and the front door logs who made it: one Information line per change with the signed-in name and role, the store, the method, the path without its query string, and the store’s status code (never the body, a token, a cookie or the query string). So give a person who must not change things a read-only role at the front door. For that person the front door sends each call with a header that makes the store answer it as a read-only caller, for that one call: the store refuses the reads it refuses a viewer and reports a viewer’s flags. The header can only lower what the session may do, and it names nobody. Every store must run a build that reads it: the front door refuses a read-only person’s call to a store that does not confirm it. Give an agent the agent role, which the front door forwards with the store’s agent-role token only. Behind a front door, every agent that uses that store reaches it with the same agent-role token, so a mute or digest rule written without a claim (see Agent Changes below) is journaled as agent:<that token's name> for all of them, and the journal cannot tell those agents apart. A change made with a claim names its agent in agent_action.agent_id, which the caller sets. A token with no name configured is journaled as agent:legacy-token. Give the front door only the tokens that its users need.
Agent Changes: List, Undo and the Switch
When a notification route is set to Level 2, an investigating agent may change monitoring settings (a mute rule, a custom alert rule, a threshold override) through the gate. Every call is journaled, so every change can be seen and taken back.
The list. The Viewer’s Investigations window has an Agent changes tab, and the web dashboard’s Admin page has an Agent changes tab. Each row is one agent call: when, server, kind, tool, agent, why and the result. A call that was journaled with neither a “done” nor a “failed” result shows result unknown: the change may or may not have happened, so it has no Undo. Below the calls, the refusals list what an agent asked for and the code it was told (for example actions_off). A mute row shows how many alerts were muted since the change.
Undo. One click restores what the setting was before the agent’s change, and only when nobody has changed it since. Undo compares the setting as it is now with what the agent left; it looks at the settings an agent can change and ignores the updated-at and updated-by stamps. If they match, the before image is restored and the undo is journaled as action_undone by person:<login>, in one transaction. If a person edited it since, Undo says changed since and leaves the setting alone. A rule the agent deleted comes back with its own Custom:<id>; if a rule with that name exists now, Undo refuses. A change the agent created is deleted; one it changed is put back. Undo works for 365 days from the change.
The switch. config_service.agent_actions_enabled is on by default: a route set to Level 2 is the opt-in. Turn off agent changes (Viewer or web) makes the gate refuse every agent change as actions_off at once; nothing already made is undone. An operator can also turn it off with set_agent_actions_enabled over MCP (mcp.allowWrites and a token, or a named operator token); enabled: true is refused and writes nothing, because an MCP client must not give agents back the actions a person took away: a person turns them on in the Viewer (Agent changes) or on the web dashboard. That tool is never offered to an agent.
Who can decide. The Viewer signs in as the store admin and can undo and switch. The web decides only under the admin store login. On the managed store the web login is derived (the web role) and postgres.webConnectionString is refused. With web.allowAdminWrites on, a person signed in to the web as an admin, which includes the shared web token, undoes and switches through the managed admin login. A new install writes it on. A darling.json without the key keeps it off, so an upgrade from 3.11 changes nothing until you set it. With it off, the page lists only and the 403 names the switch. On a bring-your-own store the 403 also says to give the web an admin store login: set postgres.webConnectionString to an admin login to decide from the web.
Server action switches (#6269). Each monitored server has a switch for each agent server action (force_plan, evict_plan, kill_session, recompile_module, create_plan_guide, drop_plan_guide), all off until a person turns one on; an agent can act on a server only while its switch for that action is on. The Viewer’s Investigations window has a Server actions tab, the web dashboard’s Admin page has a Server actions tab, and the set_server_action MCP tool turns a switch on or off. Each change is journaled as server_action_switched with who made it, from which screen, and the state before and after. The web tab turns a switch only for an admin who is signed in to the dashboard: a dashboard with no sign-in configured, or a viewer account, lists the switches and answers 403 to a change (the same rule that applies to undoing a plan force or unforce from the web). The MCP tool is served only with mcp.allowServerAdmin, takes ONE server by its exact name (a disabled server too; a part of a name is refused and the near matches are listed), and is never offered to an agent that investigates an alert. A switch cannot be turned on for an action the server’s engine cannot run: every action except kill_session is SQL Server only. A switch can always be turned off. An Azure SQL Database target cannot take evict_plan, create_plan_guide or drop_plan_guide, and an Azure SQL Managed Instance target cannot take create_plan_guide or drop_plan_guide. The edition is the one Darling collected for the server. Until Darling collects it, the switch can be turned on and the tool still refuses the call. Where none of these is at hand, the store owner can still set the row in config.config_server_actions by hand; the table’s grants are unchanged.
People, identities and grants (#7107). An administrator can name people, map each one to the identities they sign in with, and give them grants. A web identity is a signed-in web seat’s owner key, a hash of the identity provider’s issuer and subject. It is never the user name, because some identity providers reuse user names. The shared web token has no owner key, so it maps to nobody. A slack identity is the Slack team id and user id. One identity belongs to one person at most. A grant names a capability and a scope: one server, one tag, or the fleet. The capabilities are ask, request_level_2, request_level_3, approve, undo, decide_note and feedback. A request_level_3 or approve grant also lists the server action kinds it covers. The web dashboard’s Admin page has a People tab. Only a signed-in admin, or the shared web token, sees the list with each person’s sign-in keys and Slack ids. Any other session sees only its own sign-in key, to give to an administrator. MCP get_people shows the kind of each identity and never its key or ids. Only the store owner writes the three tables. The schema upgrade itself takes every write back from the four roles, on the tables and on their id sequences. So the tables are safe even before provisioning runs again after the upgrade. Removing a server or a tag removes its grants. See People, sign-ins and grants.
Only the store owner writes the three tables, as for the server action switches. The admin, viewer, web and mcp logins read them and cannot change them. Each change writes a journal row that names the administrator. Only the store owner can write those rows: person_added, person_renamed, person_enabled, person_disabled, person_identity_added, person_identity_removed, person_grant_added and person_grant_removed. The get_people MCP tool reads the directory. No MCP tool, chat message or agent changes it. Only the approve grant is read today (see “Level 3 server actions: approval” below). No other grant is checked, so administrators keep every other power.
Level 3 server actions: approval
Action approval
Each alert route has two approval settings that apply at Level 3. action_approval is required or off. A new route starts at required, so a person must approve each server action that the route lets an agent run. The upgrade to this version set off only on the routes that were at Level 3 then, so those routes keep acting as before. A route that a person raises to Level 3 later starts at required. approval_single_person starts off.
If the alerts of one request come from several routes, the request needs approval when any of those routes is required. It allows single-person approval only when every one of them does. A request that a person’s chat message opened always needs approval, whatever the route says.
What the agent gets
On a route that needs approval, a Level 3 tool called with dry_run false runs its checks and a dry run. It does not change the server. It stores a proposal and answers status proposed. The answer also holds the kind, the server, the proposal_id and nobody_can_approve. It adds what the dry run shows, such as the statement. A second call for the same request, tool and target gets the same proposal_id and adds no proposal. The answer tells the agent not to call the tool again. If a route is set to required after the gate passed, the agent’s own write is refused as approval_required.
Where proposals are stored
A proposal waits in config.agent_action_proposals for a person. Only the store owner writes that table. The admin, viewer and web logins read it. The mcp login cannot read it, because a proposal holds the statement that the action runs.
A proposal is pending until a person approves it, and it expires after 60 minutes. The service’s sweep voids a pending proposal when a person closes its request. It also voids a pending kill_session proposal when the alert of its request resolves. It marks an approved proposal unknown when no outcome arrives within 10 minutes. A plan force whose reply was lost ends the same way as the agent’s own call. The press reads the plan again, and the proposal ends as run when the plan reads as forced. If the read fails or the plan is not forced, the proposal stays unknown. Each state change writes a journal row that only the store owner can write: proposal_created, proposal_first_press, proposal_approved, proposal_run, proposal_failed, proposal_unknown, proposal_refused, proposal_expired, proposal_superseded and proposal_voided. The row proposal_notice_sent is owner-only too.
Approving a proposal
All seven action tools go through the same approval:
-
force_query_store_planandunforce_query_store_plan evict_cached_plan-
kill_session(SQL Server, Azure SQL Database and PostgreSQL) recompile_module-
create_plan_guideanddrop_plan_guide
A person approves with two presses, and the same person makes both. Each press runs the tool’s dry run again against the proposal’s server. If the facts differ from what the agent proposed, the proposal ends as superseded, nothing runs, and the answer names what changed.
The confirming press also compares the facts with the first press. A dry run that the server refuses, such as a session that is gone, ends the proposal as superseded too. A read that fails leaves it pending.
The confirming press checks the conditions that the agent’s own write checks, in one transaction:
- the service switch
- the request is not closed
- a
kill_sessionon an alert request has an alert that has not resolved - Level 3 over the live routes
- the server’s switch and the remediation login
It then checks the approver. The write runs in its own transaction, which checks the same conditions again. If they no longer hold, the proposal ends as refused with the reason. The journal records the agent as the actor of action_called and the approver in its detail. No agent tool approves a proposal. The agent’s own kill is also refused once a person closes the request or the alert of an alert request resolves (request_closed, alert_resolved).
Who can press
The approver must be an enabled person with an approve grant for that kind on that server. The grant names the server, a tag of the server, or the fleet.
- Web dashboard. A signed-in person whose
webidentity maps to such a person presses as that person. A seat that can edit and maps to nobody, such as the shared web token, presses as an administrator. A viewer-role sign-in presses only as the person it maps to. It is refused when it maps to nobody or to a disabled person. On a managed store,web.allowAdminWritesmust be on, or nobody presses on the web. - Viewer. It presses as an administrator.
- Administrator path. An administrator presses only on a route with single-person approval. On any other route the press is refused with
administrator_needs_single_person. - Asker. On a route without single-person approval, a person who asked in the request cannot approve it.
Where the statement shows, in the web page and the Viewer grid, it follows mcp.dataDetail. At maskLiterals its literals are replaced, and at metadataOnly it is left out. The agent’s reason is shown as the agent’s own words.
Nobody can approve this
The sentence “Nobody can approve this. Map a person, or turn on single-person approval.” shows in four places.
- The
proposedanswer to the agent. It adds the sentence when single-person approval is off. It also needs that no enabled person with a sign-in holds anapprovegrant for that kind that reaches the server. - The Approval column of the Viewer’s Proposals grid and the proposal notice, with the same test.
- The Viewer’s route dialog, under the approval setting. It shows when approval is
required, single-person approval is off, and no enabled person with a sign-in holds anyapprovegrant. - The web dashboard’s Admin page, on the Notification Routes tab, as “Route N: …”. It names each Level 3 route that meets three tests. Approval is not
off. Single-person approval is off. No enabled person with a sign-in holds anapprovegrant that reaches a server of the route.
For the web page see Proposals. For the grid and the dialog see Proposals in the Viewer. For the notice see Proposal notices.
Who sets approval
A person sets any value in the Viewer’s route dialog or in the web route editor. Over MCP, update_notification_route takes action_approval and approval_single_person for the operator token only, and it can only tighten them.
-
action_approvalcan go torequired, never tooff. -
approval_single_personcan go to false, never to true. -
create_notification_routetakes neither, so a route that MCP creates starts atrequiredand false. - A route with approval
offcannot be raised to Level 3 unless the same call setsrequired. A Level 3 route with approvaloffcannot be enabled, and its match, scope and severities cannot be changed. - A Level 3 route with
requiredcannot be disabled, cleared to Inherit, or have its match, scope or severities changed. Its level can still be lowered to 0, 1 or 2. - While any other enabled Level 3 route has approval
off, MCP refuses three changes to a route that has an agent level (0 to 3). It does not disable the route, clear its level, or change its match, scope or severities. Such a change can move an alert to the route with approvaloff, whose actions run with no proposal. This closes the path of lowering a Level 3 route withrequiredto Level 2 first and changing it with a second call. The refusal names both routes and says that a person makes the change in the Viewer or on the web dashboard. Settingaction_approvaltorequiredon the other route is still allowed. -
set_notification_route_enabledfollows the same rules, because on a route with an agent level it makes the same change asupdate_notification_route.
Lowering a level, raising a level (except to Level 3 on a route with approval off), and turning approval on are allowed.
get_notification_routes and every route answer show both columns. route_agent_changed journals both, before and after. The undo answer of update_notification_route leaves both out. When a call changed one of them, the answer says that a person must undo that change in the Viewer or on the web dashboard.
Known limits of approval
The MCP rules stop the plain paths. They are not a boundary against an operator token.
- An operator token can edit server tags. That can move a server out of a route’s tag scope.
- A token holder who edits server tags can move a server into or out of a tag-scoped
approvegrant. So tag edits widen or narrow who can approve. They do not skip approval, because a person with the grant must still press twice. - The shared web token is an administrator web seat. It can set
off, and it presses as an administrator on a route with single-person approval.
Level 2: agents change mute rules and custom alert rules
An alert route set to Act (Level 2) (Viewer route editor, Agent column) lets an agent that holds a live claim on a request change alert settings on that request’s server only, with mcp.allowInvestigations on, mcp.allowWrites off and a token set. A named agent token gets the same tools with mcp.allowInvestigations off. Level 2 is a per-route choice: a request opened by alerts from several routes takes the lowest level among them, and an operator can turn every agent change off with config_service.agent_actions_enabled.
Change first, report last: writes refuse after submit. The claim must be live (not lapsed, released, reported or taken by another agent id). Every change names the request, claim and agent, a reason of 1 to 2,000 characters and evidence shaped like a report’s, in the agent_action argument:
{"request_id":"...","claim_id":"...","agent_id":"agent-a","reason":"the chain is the nightly batch","evidence":[{"tool":"get_blocking","arguments":{},"key_numbers":"12 blocked"}]}
| Tool | What an agent may do | Refusal codes it adds |
|---|---|---|
create_mute_rule, update_mute_rule, set_mute_rule_enabled (on) |
Mute exactly one metric on the request’s server. The scope is server_id = the request’s server; the mute names a metric_name. The reason ends with [agent:<id> request <uuid>]. |
mute_needs_metric, self_monitor_metric, other_server |
delete_mute_rule, set_mute_rule_enabled (off) |
Delete or switch off a mute whose stored rule is on the request’s server. | other_server |
create_custom_alert_rule, update_custom_alert_rule, delete_custom_alert_rule |
Create, change, disable or delete a custom rule whose scope is exactly the request’s server, before and after. updated_by is agent:<id>. |
rule_scope |
A custom alert rule is not tied to one server by the tool call: create_custom_alert_rule and update_custom_alert_rule take no server argument, and a rule with no scope in its definition applies to every monitored server. To limit one, put {"scope":{"mode":"servers","servers":["<server name>"]}} (or {"scope":{"mode":"tag","tagId":<fleet tag id>}}) in the definition (#6455). An agent’s Level 2 rule must carry a scope of exactly the request’s server.
Mute and digest rules without a claim (#7414). An agent token can also call create_mute_rule, update_mute_rule, set_mute_rule_enabled and delete_mute_rule with no agent_action at all, for a case no request covers, such as an alert storm from a service alert. The token is a named agent token, or the unnamed token with mcp.allowInvestigations on and mcp.allowWrites off. The journal records the unnamed token as legacy-token. Each agent can make 20 of these writes an hour (agent_mute_rate). A digest here has the scope an operator has: one server or all servers, any metric including the service’s own alerts. A mute here has the claim path’s scope: it must name server_id and a metric that is not one of the service’s own alerts, and is otherwise refused with the claim path’s own reason words (mute_needs_metric, self_monitor_metric), so leaving out agent_action never widens what a mute can silence. action is mute or digest. In return the rule an agent leaves behind is bounded. expires_at is required, must be in the future and at most 7 days ahead (the refusal agent_mute_expiry names the bound), and a reason is required (agent_mute_reason). An update must carry expires_at_utc and the rule it leaves must satisfy the bound, enabling a rule needs the stored expiry to satisfy it too, and disabling or deleting a rule is always allowed. A repeat schedule (recur_*) and starts_at are refused here as they are with a claim. The service’s agent switch (config_service.agent_actions_enabled) still turns this path off, and so does mcp.allowAgentMuteRules in darling.json (default true, restart-only): set it to false and this path alone refuses with text that names the setting and writes nothing, while the claim-based Level 2 and 3 tools carry on. Three more limits keep the 7 days a lifetime. A rule must name a metric. A mute also names one server, and only a digest can cover any server. A rule with no metric is refused (agent_mute_no_metric), because it would silence every alert it covers, including the service’s own health alerts and the daily report. An operator can still make that rule. An agent cannot move a rule’s expiry later than 7 days after the rule was created (agent_mute_lifetime): it can shorten the expiry, but only an operator can extend it, and that holds for an operator’s older rule an agent updates too. An agent cannot create a rule with the same metric, server and action as one an agent made in the last 14 days, even if that rule was deleted or has expired (agent_mute_repeat, read from the agent change journal), so making a rule again does not renew it. A different metric, server or action is a different rule and is allowed. Text an agent supplies is capped at 256 characters a field (changes_json at 8192, reason at 2000), and this path allows each agent token 20 writes an hour (agent_mute_rate), which keeps a runaway agent from crowding an operator’s change off the Agent changes list. A claim-based update cannot change a digest rule or move an agent-made rule’s expiry past that lifetime. Each write adds one action_called and one action_done row to the agent change journal in the same transaction as the rule, with agent:<token name> (agent:legacy-token for the legacy token) as the actor, the reason, and the rule before and after, so Agent changes lists it and an operator can undo it. A refused or failed write leaves no rule and no row. Before the service has published its store owner connection, every verb here answers refused (owner_source_unavailable) and writes nothing. The bound applies to agent tokens only: a write with mcp.allowWrites on is unchanged. A call that carries an agent_action takes the claim path below, unchanged.
What an agent never does with a claim: it never mutes a whole server (a mute without server_id and metric_name is refused as mute_needs_metric), never mutes the service’s own alerts (a metric in the self-monitor family or one of the service’s own alert names, including Notification Channel Failing and Notification Channel Recovered, is refused as self_monitor_metric; the availability group alerts, Agent Not Running, Agent Alert Gaps and the report digests are still mutable), and never writes an enable flag, alerts_enabled, a self_alerts.* setting or an exclusion list: update_alert_settings, the notification route tools, Custom Views, server tags and the server tools are not offered to an agent at all. An agent turns an alert off only with a mute. A custom rule carries no SQL: it binds an existing view panel and a predicate and runs on the viewer-role pool. A rule’s name and description travel with its alerts through notification channels. A request opened by an alert from a custom rule an agent last wrote is Level 1.
The gate’s refusals (agent_action_required, agent_action_invalid, actions_off, claim_not_live, practice, not_level_2, other_server, journal_capped) answer {status: refused, reason, message}, change nothing and are journaled once per request, tool, code and hour. A change journals action_called (with the before image) and then action_done (with the after image) or action_failed. Under mcp.allowWrites the 9 store action tools work as before and ignore agent_action. The 7 server actions always go through the server action gate, and mcp.allowWrites never skips it. action_proposals in a report stays refused. The change, the gate’s re-check of the claim, the level and the server, and the action_done journal row are ONE transaction: a change is in force if and only if its action_done row exists, and a change that fails (a duplicate name, a stale version, a rule re-scoped since it was read) rolls back and is journaled action_failed with the answer the agent was handed. An update replaces, never stacks, the [agent:<id> request <uuid>] suffix on a mute reason. A custom rule definition that does not parse is refused as agent_action_invalid, naming the definition field.