Alert rules can now catch an instance that stops reporting entirely, not just one that crosses a threshold. Turn it on per rule, and KloudMate remembers every instance it's seen, then alerts if any of them goes silent.
An instance is a unique series identified by its labels; group a heartbeat by host_name, and you get one instance per host; group by serviceName + pod_name and you get one per pod. Each is tracked individually, with its own state, history, row on the Instances tab, and dismissal. It's off by default and opt-in per rule, so nothing changes for existing rules until you turn it on. It works with any datasource.
How it fires: an instance missing from query results moves through the normal alert lifecycle, Pending, then Firing, with the reason instance stopped reporting; last seen <timestamp>. It recovers automatically the moment it reports again. Roughly, it fires the query window plus the pending duration after the last data point, so the window should be several times your reporting interval; a 60-second heartbeat with a 60-second window will flap on one late data point; a 5-minute window absorbs the jitter.
Cleanup that doesn't require babysitting:
- Dismiss an instance you decommissioned on purpose; one click, closes right away
- Auto-close handles the rest automatically, once the configurable window passes (default 24h) while other instances are still reporting
- If the whole query goes dark instead, nothing auto-closes; every tracked instance keeps firing until data returns or you dismiss it, since a totally dark query could just mean a broken pipeline, not a dead fleet
Check out the Instance Absence Detection documentation for full details, including how it interacts with routing rules, folders, silences, and maintenance windows.