> For the complete documentation index, see llms.txt.
Skip to main content

Check out Port for yourself ➜ 

Self-healing incidents

Most teams already have alerting. Something fires when a metric crosses a threshold or a check fails. The hard part is what comes next.

Too many alerts can point at the same problem, or no single signal is enough to tell a real incident from noise. A person still has to work out what changed, decide what to do, and fix it. Then they write up what happened so it does not happen the same way again. Most of that work is manual, and quality depends on who is on call and what they remember under pressure.

The self-healing incidents pattern moves defined parts of that work to agents, inside boundaries your team sets. Your team decides which steps an agent does alone and which need a person to sign off.

Once you implement this pattern, Port scores confidence, ranks eligible responses, runs actions inside the gates you defined, and records outcomes on the incident record.

The five stages​

Port breaks incident response into five stages. Most incidents move through them in order:

  1. Detect - build confidence that something is actually wrong, not just noisy.
  2. Diagnose - work out what changed and how bad it is.
  3. Recommend - weigh a few options, not just the first idea.
  4. Act - carry out the response inside limits set in advance.
  5. Learn - turn the outcome into a change to those limits.
Self-healing incidents five-stage flow: Detect, Diagnose, Recommend, Act, Learn
Not a strict cycle

These stages are not a fixed sequence. New evidence during Diagnose can send you back to Detect. An action that fails in Act can trigger another pass through Recommend.

1. Detect​

Score confidence as signals arrive. Declare an incident only when that score clears a threshold you set in advance.

  • Several signals usually need to converge before declaration. A hard outage can be the rare exception that declares on its own.
  • Silence where telemetry should exist counts as a signal, not missing data.
  • Declaration creates an incident record with an owner and a clock, not only a note in a channel.

2. Diagnose​

Answer two questions on the incident record: what changed, and how bad is it.

  • Changes (deploys, config, feature flags, dependency shifts) are structured objects with time, owner, and scope.
  • Impact covers who is affected, severity relative to steady state, and rate of spread.
  • Both update as new evidence arrives. Diagnosis is not frozen at declaration time.

3. Recommend​

Score options before you pick one. This is decision preparation, not execution.

  • More than one candidate action is considered.
  • Each option is scored on effectiveness, risk, reversibility, and time to verify.
  • Automation eligibility filters what is in bounds before scoring starts.

4. Act​

Run the chosen response through an approval gate, with a defined rollback path. Humans provide oversight through a control loop.

  • Every action has a clear definition (inputs, scope, expected outcome, success signals, rollback) before it can run.
  • The approval gate returns autonomous, needs approval, or blocked from rules set in advance.
  • When a human is pulled in, they get a decision packet: what is known, what has been tried, and what is recommended.

5. Learn​

Turn each incident into a change to thresholds, weights, or gates, not only a closed ticket.

  • Capture expected outcome before execution and observed outcome after. Divergence is the signal.
  • Turn that divergence into a proposed change for human review.
  • An incident scorecard asks whether the system earned more, less, or the same autonomy next time, separate from how fast the incident closed.

Maturity levels​

Incident response maturity progression from Reactive (humans do all steps) to Systemic (learning runs continuously)

The five stages stay the same as you mature. What changes is how much an agent handles versus a person working from memory under pressure:

  • Reactive - humans do all five steps by hand.
  • Augmented - agents diagnose and recommend, humans act. Most teams doing this today sit here.
  • Bounded - agents act on their own for a pre-approved set of low-risk, reversible actions.
  • Adaptive - eligibility and ranking update with live confidence and impact, not only that fixed pre-approved set.
  • Systemic - learn runs often enough that incidents get rarer, smaller, and more instructive over time.

Readiness work is how teams move up that ladder: a clear catalog, ownership, scorecards, SLOs, and on-call visibility. The five stages above are what run once an incident starts. See ensure production readiness for the checks that harden services between incidents.

These guides help you implement parts of the model in Port today. More stage-specific guides will follow.

Start here

Detect

Diagnose

Act

Readiness

Communicate