Skip to content

Debounce: Requiring Consecutive Failures Before Reporting

A single noisy sample shouldn't page anyone. The consecutive-breaches setting on a threshold requires a chosen number of consecutive breaching runs before a check's confirmed severity actually reflects the breach. Every command and output below was run for real.

1. A controllable demo check

Its value comes from a table you flip between healthy and breaching, so the demonstration doesn't depend on real server load or timing:

CREATE TABLE demo_control (v numeric);
INSERT INTO demo_control VALUES (10);

CREATE OR REPLACE FUNCTION pghf.chk_demo_debounce(p_run_id uuid) RETURNS pghf.check_result
LANGUAGE sql AS $$
    SELECT ROW('COLLECTED', jsonb_build_object('value', (SELECT v FROM demo_control)), NULL)::pghf.check_result;
$$;

SELECT pghf.upsert_namespace('DEMO', 'Debounce Demo');
SELECT pghf.upsert_category('DEMO', '01', 'Debounce Demo');
SELECT pghf.create_check('DEMO', '01', 1, 'demo debounce check',
    'pghf.chk_demo_debounce(uuid)'::regprocedure, 'numeric',
    p_rationale => 'demonstrates min_consecutive_breaches',
    p_numeric_direction => 'lower_is_better');
-- => 'DEMO01-001'

2. A threshold requiring 3 consecutive breaches, then a suite to run it in

SELECT pghf.set_threshold('DEMO01-001', p_crit_value => '50'::jsonb, p_min_consecutive_breaches => 3);

SELECT pghf.create_check_run('demo_suite', 'debounce demo');
SELECT pghf.add_check_to_run('demo_suite', 'DEMO01-001');

3. Run it five times

Flip the value between healthy and breaching, using the combined collect-and-evaluate call each round:

-- round 1: healthy
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 2: breach #1
UPDATE demo_control SET v = 100;
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 3: breach #2 (still 100)
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 4: breach #3 (still 100)
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 5: recovered
UPDATE demo_control SET v = 10;
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

Pulling the whole sequence for that one check afterwards:

SELECT row_number() OVER (ORDER BY r.started_at) AS round,
       (rr.observed->>'value') AS collected_value,
       e.raw_severity, e.confirmed_severity
  FROM pghf.evaluations e
  JOIN pghf.runs r ON r.run_id = e.run_id
  JOIN pghf.raw_results rr ON rr.run_id = e.run_id AND rr.check_uid = e.check_uid
 WHERE e.check_uid = (SELECT check_uid FROM pghf.checks WHERE check_id = 'DEMO01-001')
 ORDER BY r.started_at;

 round | collected_value | raw_severity | confirmed_severity
-------+------------------+--------------+--------------------
     1 | 10               | ok           | ok
     2 | 100              | critical     | ok
     3 | 100              | critical     | ok
     4 | 100              | critical     | critical
     5 | 10               | ok           | ok

What's happening at each round:

  • Round 1 — healthy, nothing to debounce.
  • Rounds 2 and 3 — the raw severity is already critical immediately, with no smoothing on the raw signal at all, but the confirmed severity stays ok because only 1, then 2, consecutive breaching runs have happened — not yet the 3 the threshold requires.
  • Round 4 — the third consecutive critical result in a row. The confirmed severity flips to critical here, and only here.
  • Round 5 — one healthy run clears it immediately, straight back to ok. This is the default recovery behavior — instant, not symmetric with breaching. See Symmetric recovery: min_consecutive_clears below if you want that requiring N clean runs instead.

A few things worth knowing before relying on this:

  • The comparison is by severity rank, not an exact match. A warning-tier debounce is also satisfied by an intervening critical run — getting worse while waiting to confirm doesn't reset the streak. The same is true of invalid/error_in_performing_check — see How Threshold Evaluation Works in the User Guide for why those two rank above warning/critical on the full severity ladder, and what that means for a streak.
  • Debounce counts evaluated runs, not raw clock time, by default. "3 consecutive breaches" means 3 evaluated runs of the check — 15 minutes if it runs every 5 minutes, 3 days if it runs once a day. See Bounding it by wall-clock time below if that gap matters for your case.
  • One stream per check. A check's streak is tracked against the check itself — the same rule as thresholds, which are defined against the check only. If the check belongs to more than one suite, every suite's runs feed the same streak (a breach observed by one suite counts toward the streak another suite's next run confirms); that also means mixed cadences make "N consecutive" elastic in wall-clock terms, which is exactly what the time bound below pins down.
  • pghf.get_debounce_status(check_id) answers "how close is this to confirming, right now" without hand-querying pghf.evaluations — see its own section below.

Clean up the demo afterwards the same way as any other custom check — inactivate it (see Inactivating a Check), then drop the demo table and function yourself if you don't want them lingering.

Bounding it by wall-clock time: min_consecutive_window

"3 consecutive breaches" on its own says nothing about how far apart those 3 runs actually landed — a suite that misses a run, runs on an irregular schedule, or gets a manual re-trigger can stretch "3 consecutive" over a much longer real time span than you were picturing when you configured p_min_consecutive_breaches => 3. p_min_consecutive_window closes that gap: an optional interval that caps how far back (in real time) a prior run is allowed to count toward completing the streak.

SELECT pghf.set_threshold('DEMO01-001',
    p_crit_value               => '50'::jsonb,
    p_min_consecutive_breaches => 3,
    p_min_consecutive_window   => interval '10 minutes');

With this set, a prior breaching run older than 10 minutes simply isn't a candidate for the lookback at all — it's excluded outright, not counted as "too old to help." If the check's runs land every 15 minutes, this configuration can now genuinely never confirm (3 runs can't fit inside a 10-minute window if they're 15 minutes apart), and pghf.set_threshold() tells you so:

NOTICE:  set_threshold(): DEMO01-001 requires 3 breach(es) within 00:10:00, but this check's
         runs land ~00:15:00 apart on average — this may never confirm

This is purely advisory — it never blocks the call — and it only fires once there's enough run history for the check to compute an average gap from. It can't make your suite run more often either; the framework doesn't control scheduling. If you genuinely need debounce tighter than your suite's own run cadence (confirming within a window shorter than the suite realistically runs), that's a job for the check itself, not this engine-level setting — have the check function sample internally (e.g. a few pg_sleep()-spaced reads) and return one already-debounced value, rather than trying to make the outer suite outrun its own schedule.

NULL (the default) leaves this unbounded — today's original behavior, unchanged.

Windowed tolerance: min_breaches_in_window / breach_window_size

Strict-consecutive means one clean run resets the whole count back to zero, even if the overall trend is clearly real. If that's too brittle for a noisy check, windowed tolerance — "N breaches within the last M evaluated samples" — is the alternative: it doesn't need every sample to breach, just enough of them.

SELECT pghf.set_threshold('DEMO01-001',
    p_crit_value             => '50'::jsonb,
    p_min_breaches_in_window => 2,
    p_breach_window_size     => 4);

This confirms once 2 of the last 4 evaluated runs have breached — even non-consecutively (breach, clean, breach still confirms; breach, clean, clean, breach does not, since that's only 2 of the last 4, but the window has moved past the first breach by then). p_min_consecutive_breaches and this pair are mutually exclusive — pghf.set_threshold() rejects setting both (p_min_consecutive_breaches > 1 and p_breach_window_size together raise an error) — pick strict-consecutive or windowed tolerance for a given threshold, not both; there's no defined combined meaning for "exactly K in a row, within the last M" that the engine tries to guess at. p_min_consecutive_window still applies on top of either mode, unmodified.

Symmetric recovery: min_consecutive_clears

By default (p_min_consecutive_clears => 1), one clean run clears a confirmed breach immediately — round 5 in the walkthrough above. Set it higher to require that many consecutive clean runs before the confirmed severity actually drops, holding it at its last-confirmed value in the meantime rather than reporting ok early:

SELECT pghf.set_threshold('DEMO01-001',
    p_crit_value             => '50'::jsonb,
    p_min_consecutive_clears => 3);

With p_min_consecutive_breaches left at its default (1, immediate confirm), a single breach confirms right away — hysteresis only governs the recovery direction. Repeating the walkthrough with this threshold instead:

 round | collected_value | raw_severity | confirmed_severity
-------+------------------+--------------+--------------------
     1 | 10               | ok           | ok
     2 | 100              | critical     | critical
     3 | 10               | ok           | critical
     4 | 10               | ok           | critical
     5 | 10               | ok           | ok

Rounds 3 and 4 are both clean, but confirmed_severity holds at critical — the last confirmed value — until the 3rd consecutive clean run (round 5) actually clears it. This is the same rank-based, uniform comparison as the breach side: a prior invalid/error_in_performing_check confirmed value needs the same N clean runs to clear as a numeric breach would, and p_min_consecutive_window bounds the clear-side lookback exactly as it does the breach-side one.

Bounding hysteresis alone: requiring more confirmations, not more time

p_min_consecutive_window doesn't require breach debounce to be enabled — it only needs something to bound, and p_min_consecutive_clears > 1 on its own is enough. This is a deliberately supported, common monitoring shape: leave p_min_consecutive_breaches at its default of 1 (a real problem is never hidden behind an unmet streak — the check's very first breach ever still confirms immediately), while requiring several consecutive clean runs, within a bounded window, before trusting a recovery:

SELECT pghf.set_threshold('DEMO01-001',
    p_crit_value               => '50'::jsonb,
    p_min_consecutive_clears   => 3,
    p_min_consecutive_window   => interval '10 minutes');

Important: this counts evaluated runs, not elapsed wall-clock time. Nothing here enforces a minimum gap between the clean runs — it only excludes ones outside p_min_consecutive_window (see Bounding it by wall-clock time above for why that bound exists at all: scheduling jitter on unattended runs, not a deliberate cooldown). In practice, if you've just applied a real fix, you don't have to wait for the suite's schedule to see the check clear — calling pghf.run_and_evaluate('demo_suite') three times back-to-back satisfies p_min_consecutive_clears => 3 immediately, since all three land comfortably inside a 10-minute window regardless of how close together they ran. What this setting actually protects against is a single lucky/noisy sample clearing a still-real problem on a run nobody is watching — not a DBA who just fixed something and is actively re-checking it.

A window backed by neither side (p_min_consecutive_breaches = 1 and p_min_consecutive_clears = 1, the bare defaults) is still rejected — there'd be nothing for it to bound. This exact pattern — instant breach confirmation, cadence-scoped hysteresis — is what this project's own built-in checks ship with by default; see Default Thresholds Reference and the Catalog book's Suites section for the full seeded mapping (a 2x-cadence window per seeded suite, min_consecutive_clears => 2) — with the same caveat as everywhere else this cadence appears: it's a recommendation, not something this framework enforces, since it has no scheduler of its own. The seeded windows assume a check is actually run at roughly the cadence its suite is named for; if you run built-in checks on your own schedule (or purely by hand), a seeded window may be too tight or too loose for your real usage — pghf.get_debounce_status() above will tell you if the average observed gap no longer fits, and pghf.set_threshold() is how you retune it.

Checking progress without hand-querying pghf.evaluations

SELECT * FROM pghf.get_debounce_status('DEMO01-001');

Returns one row: the check's current raw/confirmed severity, which debounce mode applies (none/strict_consecutive/windowed_tolerance), how many of the required breaches (or clears) have already accumulated, whether it's actually confirmed, and — when p_min_consecutive_window is set — the check's observed average run gap (across every suite that runs it) and whether the window looks achievable at that cadence. Useful for "why hasn't this fired yet" without reconstructing the lookback query by hand. current_raw_severity/current_confirmed_severity/breach_confirmed reflect exactly what the last pghf.evaluate_run() call actually decided; the progress/achievability figures are recomputed as of right now, so they can shift slightly if called well after the check's last run and a window has since aged older data out.

Continue to Percentage-of-a-Ceiling Checks.