Alert Status¶
Everything before this chapter produces history: one row per check per run. pghf.alert_status is the current state — one continuously-maintained row per check, kept up to date by the evaluation engine, answering the three questions every monitoring integration actually asks: what is this check's status right now, since when, and when was each severity level last seen?
That query — openly readable, no grants, no joins — is the entire integration for a Grafana agent, a polling script, or any external tool. This table is the "data that underlies alerting" this framework exists to provide; what you connect to it is your choice.
What a row holds¶
| Group | Columns | Meaning |
|---|---|---|
| Identity | check_id, alert_name, check_is_retired |
Kept in step with the catalog automatically — renames, renumbers, and retirements follow. A retired check's row stays, marked, its history of timestamps intact. |
| Current stretch | current_status, current_raw_severity, current_status_first_at, current_status_last_at, current_status_repeat_count |
The confirmed severity of the latest run, when this contiguous stretch of it began, when it was last observed, and how many consecutive runs have reported it. |
| Last run | last_run_id, last_run_at, last_run_suite |
Provenance of the latest applied run — purely informational (last_run_at also tells you a check has quietly stopped running). |
| Watermarks | last_ok_at, last_info_at, last_warning_at, last_critical_at, last_invalid_at, last_error_in_performing_check_at, last_not_evaluable_at, last_skipped_at |
When each of the 8 severity levels last landed as a run's confirmed outcome. |
The current status is the literal truth¶
current_status mirrors the confirmed severity of the latest run — whatever it is. A check that self-skips shows skipped; a check whose threshold was removed shows not_evaluable. There is deliberately no open/resolved lifecycle: "resolved" is a policy judgment (is critical easing to warning resolved?), and this framework ships facts. The facts are richer than a flag: current_status = 'warning' alongside a last_critical_at twenty minutes ago is the "went critical, eased to warning" story. If you want a "resolved" concept, define it yourself on top — for example, "ok for 30 minutes with no last_critical_at in the past 2 hours" — and it will mean exactly what your operation needs it to mean.
Two refinements keep the truth honest:
- Debounce and hysteresis flow through naturally, because the row mirrors confirmed severity. During a hysteresis hold,
current_statuscorrectly stayscriticalwhilecurrent_raw_severityshows the clean observations accumulating underneath — "confirmed critical, clearing 2 of 3" from one row. - Blackouts behave by their layer. A
'run'-mode window ("we chose not to look") touches onlylast_run_*andlast_skipped_at— it can never erase a real critical stretch. An'alerting'-mode window is a real observation with the reporting silenced, so the row truthfully reads critical while no event fires; show "in alerting blackout" by joiningpghf.is_blacked_out(check_id, 'alerting')live.
A late-arriving evaluation of an older run (a crashed run evaluated after a newer one, a pg_relay run finishing out of order) writes its history row but never rewinds this table — updates only apply forward in run time.
Recipes¶
What's wrong right now, worst first — the runbook/dashboard staple (invalid and error_in_performing_check rank above warning, so broken checks surface too). pghf.friendly_time() turns the raw interval into something readable at a glance:
SELECT check_id, alert_name, current_status,
pghf.friendly_time(now() - current_status_first_at) AS for_how_long,
current_status_repeat_count
FROM pghf.list_alert_status(p_min_status => 'warning', p_include_retired => false);
Status, and the last time each check was actually ok — a single view of every evaluated check's current state alongside how long it's held that state and when it last cleared (NULL = never):
SELECT a.check_id, a.alert_name, a.current_status,
pghf.friendly_time(now() - a.current_status_first_at) AS status_duration,
a.last_ok_at,
pghf.friendly_time(now() - a.last_ok_at) AS time_since_last_ok
FROM pghf.alert_status a
ORDER BY a.check_id;
Was it worse recently?
SELECT check_id, current_status, last_critical_at,
pghf.friendly_time(now() - last_critical_at) AS critical_ago
FROM pghf.alert_status
WHERE current_status = 'warning' AND last_critical_at > now() - interval '2 hours';
The nag payload — what a notification pipeline wants to say:
SELECT format('%s is %s — %s consecutive runs, held for %s',
alert_name, current_status, current_status_repeat_count,
pghf.friendly_time(now() - current_status_first_at))
FROM pghf.alert_status WHERE check_id = 'PGHF11-009';
Checks that have quietly stopped running:
SELECT check_id, alert_name, last_run_at,
pghf.friendly_time(now() - last_run_at) AS last_ran
FROM pghf.alert_status
WHERE NOT check_is_retired AND last_run_at < now() - interval '1 day';
For Grafana specifically: point a PostgreSQL data source at the database with any login role (the table is open to read), use the first query above for a table panel, and current_status_repeat_count/now() - current_status_first_at for stat panels. No exporter, no agent-side state.
The fine print¶
Rows are created lazily at a check's first evaluation, are never deleted, and survive retention purges — pghf.purge_results_before() deletes old runs and history, but this summary keeps its watermarks (last_run_id may then reference a purged run; last_run_at still tells you when). The full stretch-by-stretch history always remains derivable from the evaluation records themselves — this table is a fast projection over that history, never a replacement for it.
Full parameter details: the Reference Guide.
Continue to Actions.