Learn from incidents
Learning from an incident well means answering two separate questions:
- What did we expect, and what actually happened?
- What should change before the next one?
Execution is where expected and observed outcomes are most explicit, but learning is not limited to that stage. Divergence can show up anywhere: a detection threshold that fired too late, a diagnosis that pinned the wrong cause, a recommendation that scored poorly, or an action that did not verify as expected.
Learning starts wherever expected and observed diverge on the incident record. Port records that divergence and can propose changes to thresholds, weights, and gates for human review.
The problem: A postmortem in a document does not change thresholds, weights, or gates on its own. The same tradeoff gets rediscovered under pressure the next time a similar pattern shows up.
The solution: Capture what was expected at each stage, compare to what happened, and turn that divergence into a proposed change:
- Measurable evidence from the incident record, not opinions reconstructed from memory.
- Concrete proposals to thresholds, weights, and gates that a human or policy can apply.
What to capture on the record
The incident record needs:
- State transitions - across detect, diagnose, recommend, and act.
- Confidence scores - tracked over time.
- Actions executed - with approval gate outcome.
- Expected outcome - what the system predicted before running an action or settling on a read.
- Observed outcome - what verification or later evidence showed actually happened.
- Control loop state - when human oversight shifted during the response.
- Human intervention - with its rationale.
If this is not captured during the incident, it is much harder to reconstruct accurately afterward. Learning depends on those expected-and-observed pairs, not just that the incident moved through each stage.
From divergence to proposal
One incident might surface a single divergence. Across incidents, the same kind of divergence may keep showing up. Examples include recommended effectiveness that keeps missing the mark, confidence scores that drift from what actually happened, or autonomous runs that end in rollback more often than expected.
When divergence or a repeating pattern crosses a threshold your team defined in advance, Port can turn that signal into a concrete proposal: adjust a scoring weight, tighten a threshold, expand automation eligibility for an action that keeps succeeding, extend a verification window for a high-variance service. Each proposal carries its justification, such as how many incidents it is based on and what the success rate looked like.
Proposals queue for human review. They do not change thresholds, weights, or gates on their own. Small, bounded adjustments can apply automatically once approved; changes that expand what runs autonomously or introduce a new action type need a human first. That split is decided in advance, parallel to the approval gate.
Incident scorecard
A fast resolution does not tell you whether the system earned more autonomy next time. An incident scorecard rates each incident across four dimensions:
- Outcome - did it reduce user impact, limit blast radius, preserve error budget.
- Decision quality - was reasoning grounded in evidence, was confidence calibrated, did recommendations arrive in time.
- Control discipline - did the system stay inside its approval gate and escalate when required.
- Learning effectiveness - did the incident produce a change to a threshold, weight, or gate.
Port scores each dimension on the incident record as healthy, constrained, or unsafe. That judgment is separate from whether the incident resolved quickly. It drives whether to expand automation eligibility and allow more autonomous execution, hold authority steady, or tighten gates and require more human review on the next incident.
When learning happens
Learning can start at any stage when expected and observed diverge, sometimes minutes into the same incident. Post-incident review consolidates what was learned; it does not create the value.
A human validates proposals already on the record: does it match intent, is the scope right, is there a plausible downside. Approved changes apply to the thresholds, weights, and gates that govern the next incident. Rejected proposals are signals too.
Back to the self-healing incidents overview.