Skip to content

Alert Status

Everything before this chapter produces history: one row per check per run. pghf.alert_status is the current state — one continuously-maintained row per check, kept up to date by the evaluation engine, answering the three questions every monitoring integration actually asks: what is this check's status right now, since when, and when was each severity level last seen?

SELECT * FROM pghf.alert_status;

That query — openly readable, no grants, no joins — is the entire integration for a Grafana agent, a polling script, or any external tool. This table is the "data that underlies alerting" this framework exists to provide; what you connect to it is your choice.

What a row holds

Group Columns Meaning
Identity check_id, alert_name, check_is_retired Kept in step with the catalog automatically — renames, renumbers, and retirements follow. A retired check's row stays, marked, its history of timestamps intact.
Current stretch current_status, current_raw_severity, current_status_first_at, current_status_last_at, current_status_repeat_count The confirmed severity of the latest run, when this contiguous stretch of it began, when it was last observed, and how many consecutive runs have reported it.
Last run last_run_id, last_run_at, last_run_suite Provenance of the latest applied run — purely informational (last_run_at also tells you a check has quietly stopped running).
Watermarks last_ok_at, last_info_at, last_warning_at, last_critical_at, last_invalid_at, last_error_in_performing_check_at, last_not_evaluable_at, last_skipped_at When each of the 8 severity levels last landed as a run's confirmed outcome.

The current status is the literal truth

current_status mirrors the confirmed severity of the latest run — whatever it is. A check that self-skips shows skipped; a check whose threshold was removed shows not_evaluable. There is deliberately no open/resolved lifecycle: "resolved" is a policy judgment (is critical easing to warning resolved?), and this framework ships facts. The facts are richer than a flag: current_status = 'warning' alongside a last_critical_at twenty minutes ago is the "went critical, eased to warning" story. If you want a "resolved" concept, define it yourself on top — for example, "ok for 30 minutes with no last_critical_at in the past 2 hours" — and it will mean exactly what your operation needs it to mean.

Two refinements keep the truth honest:

  • Debounce and hysteresis flow through naturally, because the row mirrors confirmed severity. During a hysteresis hold, current_status correctly stays critical while current_raw_severity shows the clean observations accumulating underneath — "confirmed critical, clearing 2 of 3" from one row.
  • Blackouts behave by their layer. A 'run'-mode window ("we chose not to look") touches only last_run_* and last_skipped_at — it can never erase a real critical stretch. An 'alerting'-mode window is a real observation with the reporting silenced, so the row truthfully reads critical while no event fires; show "in alerting blackout" by joining pghf.is_blacked_out(check_id, 'alerting') live.

A late-arriving evaluation of an older run (a crashed run evaluated after a newer one, a pg_relay run finishing out of order) writes its history row but never rewinds this table — updates only apply forward in run time.

Recipes

What's wrong right now, worst first — the runbook/dashboard staple (invalid and error_in_performing_check rank above warning, so broken checks surface too). pghf.friendly_time() turns the raw interval into something readable at a glance:

SELECT check_id, alert_name, current_status,
       pghf.friendly_time(now() - current_status_first_at) AS for_how_long,
       current_status_repeat_count
  FROM pghf.list_alert_status(p_min_status => 'warning', p_include_retired => false);

Status, and the last time each check was actually ok — a single view of every evaluated check's current state alongside how long it's held that state and when it last cleared (NULL = never):

SELECT a.check_id, a.alert_name, a.current_status,
       pghf.friendly_time(now() - a.current_status_first_at) AS status_duration,
       a.last_ok_at,
       pghf.friendly_time(now() - a.last_ok_at)               AS time_since_last_ok
  FROM pghf.alert_status a
 ORDER BY a.check_id;

Was it worse recently?

SELECT check_id, current_status, last_critical_at,
       pghf.friendly_time(now() - last_critical_at) AS critical_ago
  FROM pghf.alert_status
 WHERE current_status = 'warning' AND last_critical_at > now() - interval '2 hours';

The nag payload — what a notification pipeline wants to say:

SELECT format('%s is %s — %s consecutive runs, held for %s',
              alert_name, current_status, current_status_repeat_count,
              pghf.friendly_time(now() - current_status_first_at))
  FROM pghf.alert_status WHERE check_id = 'PGHF11-009';

Checks that have quietly stopped running:

SELECT check_id, alert_name, last_run_at,
       pghf.friendly_time(now() - last_run_at) AS last_ran
  FROM pghf.alert_status
 WHERE NOT check_is_retired AND last_run_at < now() - interval '1 day';

For Grafana specifically: point a PostgreSQL data source at the database with any login role (the table is open to read), use the first query above for a table panel, and current_status_repeat_count/now() - current_status_first_at for stat panels. No exporter, no agent-side state.

The fine print

Rows are created lazily at a check's first evaluation, are never deleted, and survive retention purgespghf.purge_results_before() deletes old runs and history, but this summary keeps its watermarks (last_run_id may then reference a purged run; last_run_at still tells you when). The full stretch-by-stretch history always remains derivable from the evaluation records themselves — this table is a fast projection over that history, never a replacement for it.

Full parameter details: the Reference Guide.

Continue to Actions.