Darling troubleshooting

This page describes Darling v3.13.0. The source of this page is on GitHub at that version.

Audience: operators.

This page lists the service log lines and what each one means. It moved here from the Darling operator guide.

For a table of what you see, the likely cause and the fix, start with docs/troubleshooting.md.

“Cannot load configuration” (critical, service idles) — no darling.json was found at the resolved path. The message names the path it tried; copy darling.sample.json there or point DARLING_CONFIG at your file. A file another process is holding open is retried for two minutes first, logged as "Cannot load configuration yet ... attempt N of 25" at warning; a missing, malformed or unreadable file is never retried, so this line arrives immediately for it.

“Configuration problem: …” (critical, service idles) — validation failed. The messages are literal and per-field, e.g. postgres.connectionString is required., servers must contain at least one entry., server 'X': host is required., server 'X': sql auth requires username., server 'X': sql auth requires encryptedPassword (preferred; see --encrypt-password) or password., server 'X': auth must be 'integrated' or 'sql'. Fix the file and restart the service.

“Managed Postgres bootstrap failed” — (critical, service idles) the bundled PostgreSQL could not be unpacked, initialized or started. The message names the failing step. A transiently locked file, or a cluster still coming up, is retried for two minutes first, logged as "Managed Postgres bootstrap failed, retrying ... attempt N of 25" at warning; a missing or incomplete pg-runtime.zip, a missing credential file, or an unreadable data directory is never retried, so this line arrives immediately for those. Managed mode is Windows-only.

“Cannot reach or migrate the Postgres store” (critical, service idles) — the store connection string is wrong, PostgreSQL is down/unreachable, or the login cannot create tables. Collection does not start until this succeeds; fix and restart. A transient failure does not reach this line: a store that is restarting, failing over, still coming up or at its connection limit (SQLSTATE 53300), and a sibling instance holding the migration advisory lock, are retried for two minutes first, logged as "Cannot reach or migrate the Postgres store yet ... attempt N of 25" at warning. A rung that cannot apply against this store is never retried, so this line arrives immediately for it.

“uses a plaintext password in darling.json” (warning, every connect) — you set "password" instead of "encryptedPassword". It works, but run --encrypt-password on the service machine and switch.

DPAPI decrypt fails after moving darling.json — encryptedPassword blobs are machine-bound (DPAPI LocalMachine). Re-run --encrypt-password on the new machine.

Azure SQL Database error 40615. The server firewall did not allow the IP address of the service host. Add a firewall rule for that address, or run the service on an Azure VM that reaches the server through a private endpoint. Entra authentication does not replace the firewall rule. See Azure SQL Database.

An AADSTS error on a Service Principal server. Microsoft Entra refused the sign-in. Check the tenant and the client ID. Check that the client secret has not expired. See authentication.

Error 18456, login failed for <token-identified principal>. The Entra sign-in worked, but the identity has no user in the database. Create the user with CREATE USER ... FROM EXTERNAL PROVIDER and give it the three grants. See Azure SQL Database.

“Failed to ensure XE sessions” — the login lacks ALTER ANY EVENT SESSION (or the database-scoped equivalent on Azure SQL Database). Deadlock and blocked-process collection read zero rows until the sessions exist, and client-timeout collection records a missing session; grant the permission or have an administrator create/start PerformanceMonitor_Deadlock, PerformanceMonitor_BlockedProcess and PerformanceMonitor_Attention. “Already exists / already started” XE errors are logged as benign and mean the sessions are up.

Blocked-process reports empty — the blocked-process threshold may still be 0. On AWS RDS set blocked process threshold (s) via a Parameter Group (the sp_configure bootstrap cannot run there); on Azure SQL Database the threshold is fixed at 20 seconds. Blocking stays visible either way through the always-on DMV blocking snapshot.

PERMISSIONS rows in collection_log — that collector’s reads were denied (SQL errors 229/297/300). Check the permissions; the collector retries every cycle and recovers as soon as the grant lands.

“Skipping recently-failed-job check” (info) — the login cannot read msdb.dbo.sysjobs / sysjobhistory, so failed-job alerts are skipped. Expected for minimal-privilege monitoring logins. If you want job alerts, add the direct msdb table SELECTs from the permissions section — not SQLAgentReaderRole, which gates the sp_help_job* procedures this product never calls and leaves the reads failing with error 229.

“TimescaleDB setup failed — continuing in plain-PostgreSQL mode” (warning) — the extension exists but conversion hit a problem. Everything still works (DELETE-based retention, plain tables), and nothing here waits for a restart any more. The availability half: the hourly store-maintenance tick re-probes the extension and logs “TimescaleDB is available after all” at Information when it comes back, which restores the store background-job health check (policy-job self-heal, job cadence, Retention Held) that the fallback had switched off (#3815). The conversion half: the same tick’s store-object convergence pass re-runs every ensure the failed block had reached — so the hypertables, policies and aggregates it did not get to are built on the next pass rather than at the next start (#3817).

The web viewer or MCP tools are slow — ask the store which statements it spends its time on: get_store_query_stats (MCP, or /api/read/get_store_query_stats on the web host) ranks the store’s own SQL by server-side cost from pg_stat_statements, split by the role that ran each statement, with each role’s share of the recorded time (#3899). On a managed or compose store each surface has its own login: web is the web dashboard and the service’s custom-alert rule evaluation, viewer is remote read-only Darling Viewer seats, mcp is the MCP tools, admin is the Darling Viewer on the service’s machine, and owner is the service’s collection, maintenance and alerting. A bring-your-own store runs its web dashboard and MCP server as the owner login unless postgres.webConnectionString / postgres.mcpConnectionString give them logins of their own (#3914), and then their statements rank under owner beside the service’s (the answer says so); the statements themselves still name the query. The figures are cumulative since the counters were last reset and cover server-side execution only, so a slow page whose statements rank low is spending its time elsewhere (transfer, payload size, the client). Utility statements are not tracked, so the collectors’ COPY ingest is not in them (get_collector_cost has that side). A single slow statement is also written to the store’s own log: on a managed or compose store, and on a bring-your-own store whose roles provision-roles.sql created, viewer, web and mcp statements that run past a third of their statement_timeout (5 seconds at the default 15) are logged, bind parameters excluded, and surface in get_store_log under slow_statement with comments stripped and every literal masked. The managed store loads pg_stat_statements by itself (the conf’s v13 block adds it to shared_preload_libraries, merged with anything you added there or in an included file, and turns pg_stat_statements.track_utility off so role DDL, which carries passwords, is never recorded); the preload takes effect on the store’s next service-owned start, and every start logs a warning if a later edit or an ALTER SYSTEM override drops it. On a store you run yourself, follow step 1b of tools/provision-roles.sql: add pg_stat_statements to shared_preload_libraries and set pg_stat_statements.track_utility = off (both in postgresql.conf; with ALTER SYSTEM, one quoted literal per library, 'timescaledb', 'pg_stat_statements', because a single literal holding the list is stored as one library name and the server will not start), restart PostgreSQL, and run CREATE EXTENSION pg_stat_statements; as a superuser in the darling database. If the service’s login is not a superuser, other roles’ statement text reads <insufficient privilege>: granting it pg_read_all_stats shows that text but also lets the login read every session’s query text on the cluster, and a bring-your-own store’s web and MCP hosts run as it when they have no logins of their own, so leave it out on a shared cluster. Only normalized DML shows its text; a utility statement (recorded only while track_utility is on, with its literals as typed) and a statement that opens with a comment keep their timings and read as withheld. The service builds its reader functions at its next start, and then within the hour. Until then the tool answers precondition and names what is missing and how to fix it. The store also keeps its own history of these figures: once an hour the service records each statement’s change in calls, time, rows and blocks since the last snapshot into collect.store_statement_history, the 100 statements that spent the most time, kept 90 days (#5097). Deltas survive a service restart, a store restart, a pg_stat_statements_reset() and the nightly upgrade, and each snapshot’s collect.store_statement_captures row says how many statements were active, how many entries the extension evicted, and whether the hour was a fresh baseline. get_store_query_history (MCP, or /api/read/get_store_query_history on the web host) reads it: without query_id the top statements over hours_back by time spent, with the mean of the newest quarter of their hours against the rest; with query_id that statement’s hourly series, oldest first, with the first-seen, entry-restarted and reset flags.

“Store object convergence: 14 steps, 0 changed (none), 0 failed (none), 214 ms” (information, hourly and once at startup with an at startup prefix) — the convergence pass’s own receipt (#3817). Every pass writes one, converged or not, for the reason the job-health line does: an absent line used to mean either every store object is in place or the pass never ran and nobody noticed. steps is how many ensures ran; changed names the ones that actually altered something (a converged store changes nothing, every hour, and says so); failed names any ensure that threw — each of which also logs its own warning, and each of which is retried on the next pass, which is the whole point of the cadence. A store that has just gained a collector or an aggregate shows it here once and then goes quiet. The elapsed is the pass’s own clock including connection acquisition: on a converged store it is well under a second, and a pass that takes minutes is either building something for the first time or is on a store too slow to answer its own catalog — the per-object lines above it say which.

“Store job health: 12 jobs read (compression 6, refresh 3, retention 3), 0 dead, 0 stuck, 1 held, 0 re-armed, 0 alerted” (information, hourly) — the background-job self-heal’s own receipt (#3816). Every pass writes one, healthy or not, because “no alert” used to mean either nothing is wrong or the connection open failed, was swallowed at Debug, and nothing has been checked for days. What the numbers mean:

  • dead — next_start = -infinity while not running: the scheduler has abandoned the job. Below TimescaleDB 2.26.4 the service re-arms it once (alter_job(next_start => now())) and pages; from 2.26.4 on that state is a crash backoff the scheduler clears itself, so the first sighting is logged and left alone and a second consecutive sighting pages without re-arming — re-arming there resets the backoff instead of shortening it (#3591, upstream #9360).
  • stuck — running for longer than its own bound: a run that hung and never finished.
  • held — scheduled = false, which for a retention policy is the rollup-coverage gate working as designed. Held is counted and never acted on. A held policy is not a dead one, and arming one by hand drops history its short consumer has not materialized.
  • re-armed / alerted — what this pass did about the above.

Each family pages under its own name so a mute or a notification route on one is not silently a mute on the others: Compression Job Stuck (critical — the archival tier stops draining and the disk fills), Refresh Job Stuck (critical — a rollup stops materializing, every reader of that view gets silently stale answers, and the coverage gate then holds that tier’s retention), Retention Job Stuck (warning — the store keeps history it was told to drop, which is disk and nothing else). A daily rollup that has fallen behind pages separately as Daily Rollup Stale (warning — its newest bucket is half a day from leaving its refresh window while its source holds newer rows; the job is not stuck and is not re-armed, and the text carries the refresh_continuous_aggregate call that fills the missed days; it resolves as Daily Rollup Recovered when the rollup catches up). Store Job Failing (information) is the fourth: a job whose failure count grew since the previous hourly sample and whose last run reports Failed — alive and being retried, but failing, which no other surface reported (#2136’s cadence check reads only successful runs, so it gets quieter as a job gets sicker).

“out of background workers” / “failed to start a background worker” in the postmaster log, or the store keeps growing despite compression — bring-your-own stores only: the cluster has fewer worker slots than the store has policies, so compression and retention jobs are being skipped. An occasional one is benign (the job retries on its next schedule); persistent ones mean the store is effectively uncompressed. Size the two settings and restart the server — see Background workers, and multiply them if the cluster hosts more than one store. timescaledb_information.job_stats tells you whether jobs are actually succeeding.

“Why are there 40+ postgres.exe processes?” — the count is three populations, and only one is client connections: (1) PostgreSQL’s own system processes (postmaster, checkpointer, WAL/background writers, autovacuum, stats); (2) TimescaleDB background workers — the managed conf sizes timescaledb.max_background_workers from host RAM (74 on a 4 GB host, up to 177 on 12 GB or more), and every RUNNING compression/retention policy job is its own process, so the count legitimately surges during checkpoint/compression waves and falls back when they finish; (3) client backends — the service’s pools are capped at 24, the co-located viewer’s at 10. Decompose it live with: SELECT backend_type, count(*) FROM pg_stat_activity GROUP BY backend_type ORDER BY 2 DESC; — and remember Windows charges the shared buffer segment to every attached process’s working set, so per-process memory numbers cannot be summed.

query_store bursts every ~15 minutes — two or three near-empty cycles, then one large one, is Query Store’s own behavior, not a collector bug: the engine buffers in memory and flushes to its persisted tables on DATA_FLUSH_INTERVAL_SECONDS (default 900s), so the collector genuinely sees nothing new between flushes. Narrowing the collection interval will not smooth it. The per-database log lines show which database drove a burst.

MCP client cannot connect — MCP defaults to off. Enable it live from the Viewer’s Settings (the checkbox writes the control plane; the service starts the endpoint within seconds, no restart), or set mcp.enabled: true in darling.json for a file-seeded install. If the log says Port 5152 is already in use — MCP server not started, change mcp.port. The MCP server binds to localhost only unless you opt into a LAN endpoint (see Opt-in Network Endpoints (LAN)); a remote client that gets 401 is missing or mismatching the required bearer token, and one that is refused before any response is outside the configured allowFrom CIDR.

Recommendations tab says no findings — analysis runs every 30 minutes per server but only once the store holds at least 24 hours of collected data for that server; a fresh install simply has not earned findings yet. (The exception is same-statement pileup, which needs only the current snapshot plus 45 minutes of that statement’s own history, so it can appear on a young install while every other finding is still waiting out the data-span gate.)

The Viewer will not connect — the failure window carries a Configuration this viewer used block naming the darling.json it read (and which rule picked it: an explicit command-line path, DARLING_CONFIG, beside the viewer, or the service root), plus the host, port, username, database, SSL mode, search path and certificate path it parsed. Read it before changing anything: the two faults it separates are the viewer read a different file than you edited and it read your file and a value in it is wrong. Copy details puts the whole block on the clipboard for a bug report, and the same lines are in %APPDATA%\PerformanceMonitorDarling\logs\darling-viewer_yyyyMMdd.log. It never contains a password.

“Root Certificate … exists: NO” — with SSL Mode=VerifyFull, a relative Root Certificate path resolves against the folder holding the darling.json the viewer read — not the working directory, so how the viewer was launched no longer changes the answer. The diagnostics block prints that folder and the absolute path it actually opened; either put server.crt beside the config or make the Root Certificate value an absolute path. If you no longer have the certificate, re-run --export-viewer-config on the store host and copy the folder again (see Connect a Remote Viewer) — it regenerates if the bind IP changes, so an old copy stops matching.