Detect incidents
Detecting an incident well means separating two questions that traditional alerting collapses into one:
- Does the evidence look unusual?
- Are you confident enough to call it real?
A threshold crossing answers neither on its own. It works for failures loud enough to spike a single metric, but misses partial outages, slow-building degradation, or a fleet that goes quiet instead of throwing errors.
In most setups, detection happens upstream: monitoring, on-call platforms, or open-source alert pipelines fire the first signal. Port acts as the system of record. It aggregates those signals, scores confidence where you configure it, and governs when they become a declared incident the rest of the response can build on. Port can sit earlier in the pipeline, but that is not how most teams run it today.
The problem: Static thresholds treat one noisy metric the same as a confirmed, multi-signal event. That means either you get paged on noise, or you miss the incident that never crosses a line, because it shows up as silence instead of a spike.
The solution: Score confidence from multiple signals as they arrive, and declare only once that score clears a threshold your team set in advance. This gets you:
- Earlier detection of silent failures, because confidence builds from a source going quiet, not from an error rate crossing a line.
- Fewer false positives, because declaration usually needs several signals converging. A genuinely unambiguous signal (a hard outage, a payment processor failing every request) is the rare exception that can declare on its own.
- Confidence that decays, so if fresh evidence stops arriving, the score drops back down instead of staying pinned on a stale read.
- A record of why, not just that, so responders get the specific signals and the leading hypothesis behind a declaration, not a bare page.
How confidence gets scored
Three inputs feed the score as evidence arrives:
- Signal strength - how abnormal the reading is: a spike, a degradation, or silence where telemetry should exist.
- Convergence - whether independent signals point at the same explanation.
- Persistence - whether the pattern holds as new evidence arrives, or fades after a single blip.
Silence is evidence
A source that goes quiet when it should be reporting is a signal, not missing data. Score it the same way you would an anomalous reading, and watch the expected cadence of a source, not only its content.
Declaration creates an incident record
Once confidence clears its threshold, declaration creates a governed state change: an incident record with an assigned owner and a clock. The rest of the response builds on that record. A message in a channel has neither, and it does not stop anyone from quietly deciding it was not a real incident without that decision going on the record.
Each later stage feeds its results into the incident record this step created: diagnose, recommend, act, and learn.
Next: once an incident is declared, the next question is what changed and how bad it is. See Diagnose incidents.
Back to the self-healing incidents overview.