Skip to content

Debounce: Requiring Consecutive Failures Before Reporting

A single noisy sample shouldn't page anyone. The consecutive-breaches setting on a threshold requires a chosen number of consecutive breaching runs before a check's confirmed severity actually reflects the breach. Every command and output below was run for real.

1. A controllable demo check

Its value comes from a table you flip between healthy and breaching, so the demonstration doesn't depend on real server load or timing:

CREATE TABLE demo_control (v numeric);
INSERT INTO demo_control VALUES (10);

CREATE OR REPLACE FUNCTION pghf.chk_demo_debounce(p_run_id uuid) RETURNS pghf.check_result
LANGUAGE sql AS $$
    SELECT ROW('COLLECTED', jsonb_build_object('value', (SELECT v FROM demo_control)), NULL)::pghf.check_result;
$$;

SELECT pghf.upsert_namespace('DEMO', 'Debounce Demo');
SELECT pghf.upsert_category('DEMO', '01', 'Debounce Demo');
SELECT pghf.create_check('DEMO', '01', 1, 'demo debounce check',
    'pghf.chk_demo_debounce(uuid)'::regprocedure, 'numeric',
    p_rationale => 'demonstrates min_consecutive_breaches',
    p_numeric_direction => 'lower_is_better');
-- => 'DEMO01-001'

2. A threshold requiring 3 consecutive breaches, then a suite to run it in

SELECT pghf.set_threshold('DEMO01-001', p_crit_value => '50'::jsonb, p_min_consecutive_breaches => 3);

SELECT pghf.create_check_run('demo_suite', 'debounce demo');
SELECT pghf.add_check_to_run('demo_suite', 'DEMO01-001');

3. Run it five times

Flip the value between healthy and breaching, using the combined collect-and-evaluate call each round:

-- round 1: healthy
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 2: breach #1
UPDATE demo_control SET v = 100;
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 3: breach #2 (still 100)
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 4: breach #3 (still 100)
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

-- round 5: recovered
UPDATE demo_control SET v = 10;
CALL pghf.run_and_evaluate(p_run_key => 'demo_suite');

Pulling the whole sequence for that one check afterwards:

SELECT row_number() OVER (ORDER BY r.started_at) AS round,
       (rr.observed->>'value') AS collected_value,
       e.raw_severity, e.confirmed_severity
  FROM pghf.evaluations e
  JOIN pghf.runs r ON r.run_id = e.run_id
  JOIN pghf.raw_results rr ON rr.run_id = e.run_id AND rr.check_uid = e.check_uid
 WHERE e.check_uid = (SELECT check_uid FROM pghf.checks WHERE check_id = 'DEMO01-001')
 ORDER BY r.started_at;
 round | collected_value | raw_severity | confirmed_severity
-------+------------------+--------------+--------------------
     1 | 10               | ok           | ok
     2 | 100              | critical     | ok
     3 | 100              | critical     | ok
     4 | 100              | critical     | critical
     5 | 10               | ok           | ok

What's happening at each round:

  • Round 1 — healthy, nothing to debounce.
  • Rounds 2 and 3 — the raw severity is already critical immediately, with no smoothing on the raw signal at all, but the confirmed severity stays ok because only 1, then 2, consecutive breaching runs have happened — not yet the 3 the threshold requires.
  • Round 4 — the third consecutive critical result in a row. The confirmed severity flips to critical here, and only here.
  • Round 5 — one healthy run clears it immediately, straight back to ok. Recovery is not symmetric with breaching: it doesn't take 3 good runs to clear, just 1 — a deliberate simplification worth knowing about if your own experience with the framework suggests it should behave differently.

A few things worth knowing before relying on this:

  • The comparison is by severity rank, not an exact match. A warning-tier debounce is also satisfied by an intervening critical run — getting worse while waiting to confirm doesn't reset the streak.
  • Debounce counts evaluated runs, not clock time. "3 consecutive breaches" means 3 runs of whichever suite is being evaluated — 15 minutes if you run every 5 minutes, 3 days if you run once a day. There's no independent time-window option.
  • Scoped to the specific check-and-suite combination. The same check running in two different suites debounces independently in each — a streak in one doesn't count towards, or interfere with, the other.
  • Works identically for a per-suite override (see Overriding a Threshold Per Suite) — the consecutive-breaches setting is just another field on whichever threshold record applies, default or override included.

Clean up the demo afterwards the same way as any other custom check — inactivate it (see Inactivating a Check), then drop the demo table and function yourself if you don't want them lingering.

Continue to Overriding a Threshold Per Suite.