Skip to content

How Threshold Evaluation Works

Threshold evaluation is a separate, optional layer that judges already-collected data. Nothing here touches collection — the evaluation engine never re-runs a check, it only reads raw results that a collection pass has already recorded.

The severity scale

Four ascending alert levels, matching the same model used by the upstream project this framework's default thresholds are drawn from:

ok  <  info  <  warning  <  critical

Plus four more states, all ranked (via pghf._severity_rank()) but not part of that alert-level progression itself:

  • skipped — the check reported itself as skipped (self-authored, or an engine-level version/topology exclusion). Ranks below ok — a deliberate non-event never counts toward, or interferes with, a debounce streak.
  • not_evaluable — the check collected fine, but no threshold is configured to judge it against. Also ranks below ok. Many jsonb-typed checks (a structured breakdown with no single number to threshold) fall here permanently; others are simply checks nobody has set a threshold for yet.
  • invalid — the check ran and returned data, but a custom evaluator or the default comparator raised an exception trying to judge it (a malformed value, an unsatisfiable comparison). Ranks above warning — deliberately, not a bug: something that can't be sensibly judged is treated as at least as urgent as a real breach, not quietly discarded.
  • error_in_performing_check — the check itself raised an exception during collection; it never produced data at all. Ranks above critical, the highest rank of all — "we couldn't even check" is treated as more urgent than any known breach.

The full ascending order, then, is:

skipped  <  not_evaluable  <  ok  <  info  <  warning  <  invalid  <  critical  <  error_in_performing_check

Two consequences of ranking invalid/error_in_performing_check above the real alert tiers, both deliberate: a prior invalid/error_in_performing_check run inside a debounce lookback can count toward confirming a warning/critical streak (same uniform rank comparison used everywhere), and an event with a warning/critical floor can fire off a check that starts returning invalid data or erroring outright — see Events — where the old single collapsed unknown state never fired anything for a broken check.

Setting a threshold

One function, pghf.set_threshold(), is the single entry point for every case: seeding a built-in check's threshold, adding one for your own custom check, or updating either later. A threshold is defined against the check only — exactly one per check, applying wherever the check runs; suites decide what runs together, checks own what the results mean. It's an upsert — there is no override concept and no protected, unchangeable state:

-- the check's one threshold (applies to every suite that runs this check)
SELECT pghf.set_threshold('PGHF01-003', p_warn_value => '75'::jsonb, p_crit_value => '90'::jsonb);

-- update a seeded value later — same function, replaces it in place
SELECT pghf.set_threshold('PGHF01-003', p_warn_value => '70'::jsonb, p_crit_value => '85'::jsonb);

(If you genuinely need two different sensitivities for the same metric, copy the check — Copying an Existing Check — and threshold the copy independently.)

Any combination of the info, warn, and crit tiers may be set — many real checks only use one or two of them. Boolean (true/false) checks use a single "which severity does a failing value map to" parameter instead of numeric tiers, since there's no scale to set tiers along. Beyond the plain scalar case, the same function also thresholds one number inside a multi-metric (jsonb) check's payload (p_value_path), matches a text check's value against a pattern (p_text_pass_pattern), and judges the change since the previous sample instead of the value itself (p_delta_kind) — see Adding a Threshold for all three.

Evaluating a run

CALL pghf.evaluate_run(p_run_id => '<run_id>'), after collection has already run, judges that run's data. See Running It End to End for a full worked example against the seed catalog, with real captured output.

pghf.evaluate_run() is a PostgreSQL procedure, not a function — procedures can't hand back a result set directly, only individual output values. Two separate, read-only functions cover reading the results back out, for two different audiences. They are also the only supported way to read this data — the underlying tables aren't openly readable (see The Access Model below), so these two functions are the actual interface:

pghf.get_results_json(p_run_id) — one block of structured data, for a machine to consume. This is what you'd call to build the payload for an external notifier, a webhook, or a dashboard backend — anything that wants "the results of this run" as one self-contained unit and will do its own filtering:

SELECT pghf.get_results_json('<run_id>');
-- [{"check_id": "PGHF01-003", "raw_severity": "ok", "confirmed_severity": "ok", "message": null, "detail": {"value": 1.03}}, ...]

A flat list, one entry per evaluated check, ordered by check ID. No severity filtering — it's the complete picture, and it's up to whatever consumes it to decide what matters.

pghf.get_results_sql(p_run_id, p_min_severity => NULL) — a proper result set, for a database administrator reading output directly in psql:

SELECT * FROM pghf.get_results_sql('<run_id>');
--  severity | category_id |  check_id  |                  check_name                   | value  |                  message
-- ----------+-------------+------------+-----------------------------------------------+--------+--------------------------------------------
--  critical | PGHF11      | PGHF11-009 | Superuser login roles as % of all login roles | 100.00 | value 100.00 exceeds critical threshold 50
--  ok       | PGHF01      | PGHF01-003 | Connection saturation                         | 1.03   |
--  ...

-- narrow to just what needs attention:
SELECT * FROM pghf.get_results_sql('<run_id>', p_min_severity => 'warning');   -- warning + critical only

Worst severity first, already joined so the check's name and its actual collected value are right there — no second query needed. p_min_severity works like a logging level: the tier you name, and everything at or above its rank, is included. The default (leave it unset) shows literally everything, regardless of rank. Asking for 'info', 'warning', or 'critical' progressively narrows the result — and note 'warning'/'critical' also pull in 'invalid'/'error_in_performing_check' rows, since those rank above them (see the severity scale above).

Both functions work for any run ID, not only one you just evaluated in the same session, and both raise a clear error for a run ID that doesn't exist at all — rather than silently returning something empty. A run that exists but simply hasn't been evaluated yet is a different, legitimate case: that's an empty answer, not an error.

Knowing whether a run actually finished

Neither function above tells you whether a run actually completed. Because the collection engine commits after every single check (so a crash partway through never loses what was already collected), an interrupted run's partial results look entirely plausible on their own — nothing in the results distinguishes "genuinely nothing critical happened" from "the run crashed before the one check that would have found something critical ever ran." pghf.get_run_status() closes that gap:

SELECT * FROM pghf.get_run_status('<run_id>');
--     ...      | finished_at | is_collection_complete | checks_expected | checks_observed | checks_evaluated | is_evaluation_complete
-- --------------+-------------+-------------------------+------------------+------------------+-------------------+-------------------------
--     ...       | 2026-...    | t                       | 160              | 160              | 160               | t

is_collection_complete simply means the run has a finish time recorded; is_evaluation_complete means every collected check has also been evaluated. checks_expected reflects the suite's current membership, not what it was at the time of that run — suite membership isn't tracked historically — so for an older run, treat this as a sanity check rather than something that must match exactly.

Finding a run in the first place

Everything above assumes you already have a run ID. Finding one is pghf.list_runs()'s job — your entry point into all of this when you don't have direct access to the underlying tables (see The Access Model below):

-- what result sets are even available? most recent first
SELECT * FROM pghf.list_runs();

-- narrow to one suite, or a time window
SELECT * FROM pghf.list_runs(p_run_key => 'seeded-1h', p_since => now() - interval '7 days');

This returns one row per run — which suite it was, when, and a count of each result type, both from the collection side (how many collected, skipped, or errored) and the evaluation side (how many at each severity — all zero for a run that hasn't been evaluated yet, which isn't an error). A limit parameter (defaulting to the 50 most recent, unbounded if you pass none) caps how far back it looks.

Once you have a run ID from this, a handful of further read-only functions give you the full detail behind it: the plain run record itself (who or what triggered it, the server's PostgreSQL version, whether it's part of a Spock cluster); every check's full collection detail rather than just its evaluated value (useful for a jsonb-typed check, or for reading a skipped check's explanation); and the extra parameters a run was given (useful for working out why a check that depends on one of them reported itself as skipped). All of these are documented in full, with their exact parameters, in the Reference Guide.

Debounce — requiring more than one bad result in a row

A threshold's debounce setting (min_consecutive_breaches, default 1, meaning no debounce at all) requires a chosen number of consecutive breaching runs before a check's confirmed severity actually reflects the breach — its raw severity always shows exactly what that one run's data says on its own, with no smoothing. Two variants exist:

  • Strict-consecutive (min_consecutive_breaches > 1) — every one of the required runs must breach, no exceptions; one clean run resets the count to zero.
  • Windowed tolerance (min_breaches_in_window/breach_window_size) — "N breaches within the last M evaluated samples," tolerant of an occasional clean run in between. Mutually exclusive with strict-consecutive.

Either mode can optionally be bounded by wall-clock time (min_consecutive_window) — an interval capping how far back a prior run is allowed to count toward the streak, so "3 consecutive breaches" can mean "3 within an actual 10-minute window," not just "the last 3 evaluated runs regardless of how far apart they landed." pghf.set_threshold() warns (advisory NOTICE, never blocking) if the configured window looks unachievable given the suite's own observed run cadence.

By default, a single healthy run always clears a confirmed breach immediately — recovering isn't held to the same "N in a row" rule as breaching is. Setting min_consecutive_clears (default 1) higher makes recovery symmetric: the confirmed severity holds at its last-confirmed value across clean runs until that many consecutive clean runs accumulate, rather than dropping to ok on the first one. Without it, a check flapping right at its threshold will alert, clear, alert, clear on every other run once past its first confirmation.

The lookback for all of this is one stream per check: every evaluated run of the check counts toward its streak, whichever suite produced it — consistent with thresholds themselves being defined against the check only. If a check runs in more than one suite, or on more than one cadence, "N consecutive" means N consecutive evaluations regardless of spacing — which is exactly why the wall-clock bound above exists ("3 within 10 minutes" stays stable no matter who runs the check). pghf.get_debounce_status(check_id) reports a check's current progress toward confirming or clearing without hand-querying pghf.evaluations yourself.

See Debounce in Adding Your Own Checks for a full worked example of every option above.

Custom evaluators

For a condition a flat threshold simply can't express — a rolling average, a rule that depends on more than one field at once, a categorical judgment on text — register a function instead of numeric tiers. This uses the same extensibility idea as checks themselves: write a function with the right shape, and the framework calls it automatically, with no special-casing needed anywhere else. Seven such functions ship with the framework already, covering things like a self-adjusting rolling baseline for WAL growth rate, a rule that treats replication lag differently depending on whether a standby is synchronous, a rule that treats a dead subscription worker as more severe than one that's merely logging apply errors, and a categorical judgment for a text result that has no default scalar comparison path at all. See Writing a Custom Evaluator for the full pattern and worked code.

Continue to The Access Model.