侠Liangshen AIailiangshen.com
AI Implementation Workshop · Short topic lessons · Lesson 12

Incident Post-Mortem and Alert Consolidation

Convert duty logs into a timeline and consolidate alerts into a few priority items.

Freeabout 10 minNo experience neededWeb lesson + slides + video

This lesson covers incident post-mortems and alert consolidation. We will do two things: organize messy duty logs into a timeline and draft an initial post-mortem; consolidate hundreds of alerts into groups and select a few key items for human review.

What this lesson covers

  • Scenarios
  • Methods
  • Prompts and Examples
  • Mini-Exercises, Quizzes, and Takeaways

01

Scenarios

About 1 minutes

Let's look at a scenario firstFictional example; numbers are illustrative

Hong Zhiyuan, Network Operations Engineer at Duchuan Communications

Each shift faces hundreds of alerts, with repeated alerts from the same source.

Shift handovers rely on handwritten notes, taking about 70 minutes per shift.

Qiu Ziang, Backend Developer at Qilan Intelligence, dealt with an online incident last night. The post-mortem requires documenting the sequence, impact, root cause hypotheses, and improvement items, taking about three hours.

Common mistakes: incorrect timeline order, treating hypotheses as confirmed causes, and improvement items lacking assigned owners.
02

Methods

About 2 minutes

Method: Five Steps, Read-Only, No Action

1
Provide only de-identified timestamps and phenomenaReplace device names with 'Device 01', customers with 'Customer A'; do not include full logs, source code, network addresses, or user content
2
Sort by time and mark gapsIf the record does not show who was on duty or what actions were taken during a specific period, mark it as a gap
3
Separate facts from hypothesesAll hypotheses must be marked 'To Be Verified' and must not be included in the 'Root Cause' section
4
Consolidate alerts into groupsIdentify three key items for human review, count the frequency of each alert type, and prevent low-priority but frequently recurring alerts from being overlooked
5
Confirmed by a humanThe cause is written by a person; the impact numbers are verified against monitoring; improvement items include the responsible person and deadline.
It only reads and does not take action; merging is just a suggestion, and whether to create a ticket is decided by the shift supervisor.

Check each item before posting

First, review the company's AI usage policy. Source code, keys, and customer network data must not be posted in public AI applications; the materials for this exercise are fictional and can be posted directly.
03

Prompts and Examples

About 4 minutes
Tip

Prompt 1: Alert Merging, Read-Only, No Action

Suggested time
Prompt 1 (Alert Merging)
You are an operations shift supervisor assistant with ten years of experience. You only read and do not take action.
Below are the alerts for a shift (already de-identified: devices are written as Device 01, etc.).
Please:
1. Merge them into groups based on likely common root causes, and list the time and quantity of items in each group;
2. List the top three items worth reviewing first, and explain why;
3. Count the number of occurrences for each type of alert;
4. End with 'Information I did not see.'
Rules: Only merge; do not determine severity levels or provide any action commands. For alerts not provided to you, do not write 'No anomalies.' Use only the content I provide; if something is missing, write 'To Be Supplemented,' and do not make things up.
Alerts: [Paste]
Groups must be traceable back to the original alerts for verification; any unseen information should be listed at the end.
Tip

Prompt 2: Post-Mortem Draft, Separate Facts and Speculation

Suggested time
Prompt 2 (Post-Mortem Draft)
Below is the time-and-phenomenon record of an online incident (already de-identified).
Please:
1. Sort by time and mark time gaps: which periods have no record of what anyone did;
2. Divide into two columns: the 'Facts' column contains only the time and phenomena explicitly stated in the records; the 'Speculation' column contains reasons guessed by people in the records, with each item marked 'To Be Verified';
3. Draft the post-mortem structure: Process, Impact (numbers to be verified by me), Cause (write only 'To Be Confirmed'), Improvement Items (write 'To Be Confirmed' for responsible person and deadline);
4. End with 'Information I did not see.'
Rules: Do not write speculation in the Cause section; do not fabricate times.
Record: [Paste]
The Cause column only writes 'To Be Confirmed'; all guesses are in the Speculation column and marked 'To Be Verified.'

Example: Organizing an incident record into a timelineFictional example; numbers are illustrative

Original Record

21:40 Version 2 deployment completed; 21:52 Response time increased; 22:05 Someone in the group chat said 'It might be that the database connections are full'; 22:30 Decision to roll back; 22:41 Response time returned to normal.

Organized

  • Facts Column: Four time points: 21:40, 21:52, 22:30, 22:41
  • Speculation Column: The comment at 22:05, marked 'To Be Verified'
  • Time Gap: Between 21:52 and 22:05, there is no record of what anyone did
  • Reason: To be confirmed
22:30 Why was the rollback decided? The record provides no basis for this decision. This is a gap that a person must fill.

Common Biases and Remedies

Mistaking hypotheses for causes

Remedy: The Cause section should only state 'To be confirmed.' Move all sentences containing words like 'possibly' or 'probably' to the Hypotheses section.

Overlooking low-priority but recurring alerts

Remedy: Instruct the system to count the occurrences of each alert type. Even if an alert is low-priority, it should be listed if it occurs frequently.

It states 'All other alerts are normal'

Remedy: Require it to report the number of alerts reviewed versus the total number. Randomly check a few samples. If the numbers do not match, it has not reviewed all alerts.

It provides handling suggestions or commands

Remedy: Remind it that the role is read-only for handling; delete any commands. Decisions on handling are made by the Shift Lead.

If improvement items lack an owner and deadline, mark them as 'To be confirmed.' Do not allow the system to assign them itself.
04

Mini-Exercises, Quizzes, and Takeaways

About 3 minutes

Three quick questions to check yourself

Question 1: Who writes the 'Cause' in the post-mortem?

Answer B.Dragon Warrior can only provide hypotheses; causes must be confirmed by a person, and all hypotheses must be marked 'Pending Verification'.

Question 2: It states 'All other alerts are normal.' What do you do?

Answer B.Alerts that have not been reviewed cannot be marked as 'No anomalies.' If the numbers do not match, it has not finished reviewing.

In this lesson, take away four key phrases.

  1. Only include the timestamp and the incident description in de-sensitized data; do not post source code, keys, or customer logs.
  2. Sort the timeline chronologically and mark any time gaps.
  3. Separate the facts column from the speculation column; label all speculation as 'pending verification'.
  4. Treat alerts as read-only; do not write 'no anomaly' for items you haven't read.
Show the full transcript (video narration)
Incident Post-Mortem and Alert Consolidation

This lesson covers incident post-mortems and alert consolidation. We will do two things: organize messy duty logs into a timeline and draft an initial post-mortem; consolidate hundreds of alerts into groups and select a few key items for human review. First, let's look at a scenario.

Let's look at a scenario first

Let's look at a scenario, which is fictional, with illustrative numbers. Hong Zhiyuan, a Network Operations Engineer at Duchuan Communications, faces hundreds of alerts per shift, with repeated alerts from the same source. Shift handovers rely on handwritten notes, taking about 70 minutes per shift. Qiu Ziang, a Backend Developer at Qilan Intelligence, dealt with an online incident last night. The post-mortem requires documenting the sequence, impact, root cause hypotheses, and improvement items, taking about three hours. Both often make similar mistakes: incorrect timeline order, treating hypotheses as confirmed causes, and improvement items lacking assigned owners.

Method: Five Steps, Read-Only, No Action

The process has five steps. Step 1: Provide only the de-identified time and phenomenon. Write 'Device 01' for devices and 'Customer A' for customers. Do not paste the entire log, source code, or network addresses. Step 2: Sort by time and mark time gaps. Step 3: Divide into 'Facts' and 'Speculation' columns; all speculation must be marked 'To Be Verified.' Step 4: Merge alerts into groups, list the top three items worth reviewing first, and count the occurrences of each type. Step 5: A human confirms: the cause is written by a person, numbers are verified against monitoring, and improvement items include the responsible person and deadline. It only reads and does not take action; merging is just a suggestion.

Check each item before posting

Before posting, please check each item. First, the content contains only time and phenomena, with no source code, keys, or customer logs. Second, devices, customers, and addresses have all been replaced with codes. Third, I know the reasons it provides are only hypotheses to be verified. Fourth, I will cross-reference the original records and spot-check two merging results. Additionally, a reminder: review the company's AI usage policy first. Source code, keys, and customer network data must not be posted in public AI applications. The materials for this exercise are fictional and can be posted directly.

Prompt 1: Alert Merging, Read-Only, No Action

The first prompt is for alert merging, read-only, and no action. It merges alerts based on likely common root causes, lists the top three items worth reviewing, counts the occurrences of each type of alert, and ends with 'Information I did not see.' The rules are strict: do not determine severity levels, do not provide action commands, and for alerts not provided to it, do not write 'No anomalies.' If something is missing, write 'To Be Supplemented,' and do not make things up.

Prompt 2: Post-Mortem Draft, Separate Facts and Speculation

The second prompt is for the post-mortem draft. First, sort by time and mark time gaps; then divide into 'Facts' and 'Speculation' columns, with each speculation item marked 'To Be Verified'; then draft the post-mortem structure, writing 'To Be Confirmed' in the Cause column and for the responsible person and deadline of improvement items. There are only two rules: do not write speculation in the Cause section, and do not fabricate times.

Example: Organizing an incident record into a timeline

Consider a fictional incident record with illustrative numbers. The original record contains five entries: 21:40 – Version 2 release completed; 21:52 – Response time increased; 22:05 – A team member suggested the database connections might be exhausted; 22:30 – Decision to rollback; 22:41 – Response time returned to normal. After organizing the data, the Facts section lists four timestamps, while the statement about database connections is moved to the Hypotheses section, marked as pending verification. There is no record between 21:52 and 22:05, creating a time gap. The reason for the rollback at 22:30 is not documented, requiring human input to fill the gap.

Common Biases and Remedies

There are four common biases. First, mistaking hypotheses for causes; the remedy is to mark the Cause section only as 'To be confirmed.' Second, overlooking low-priority but recurring alerts; the remedy is to have it count occurrences and list those that appear frequently. Third, it states 'All other alerts are normal'; the remedy is to require it to report the number of alerts reviewed and randomly check a few samples—if the numbers do not match, it has not finished reviewing. Fourth, it provides handling suggestions or commands; the remedy is to remind it that the role is read-only for handling and delete any commands. If improvement items lack an owner and deadline, mark them as 'To be confirmed' and do not allow the system to assign them itself.

Quiz Question 1

Let's take the quiz. Question 1: Who writes the 'Cause' in the post-mortem? A. Dragon Warrior writes it directly, as it looks the most professional. B. A person confirms it; Dragon Warrior can only provide hypotheses, which must be marked 'Pending Verification'. C. The Shift Lead writes it based on impression, then has Dragon Warrior polish it. The answer is B. Dragon Warrior can only provide hypotheses; causes must be confirmed by a person, and all hypotheses must be marked 'Pending Verification'.

Quiz Question 2

Question 2: It states 'All other alerts are normal.' What do you do? A. Accept it directly and include it in the handover document. B. Require it to list the number of alerts reviewed and randomly check a few samples. C. Write 'No anomalies' in the improvement items. The answer is B. Alerts that have not been reviewed cannot be marked as 'No anomalies'; if the numbers do not match, it has not finished reviewing.

In this lesson, take away four key phrases.

Let's review: take away these four phrases from this lesson. First, only include the timestamp and incident description in de-sensitized data; do not post source code, keys, or customer logs. Second, sort the timeline chronologically and mark any time gaps. Third, separate the facts column from the speculation column; label all speculation as 'pending verification'. Fourth, treat alerts as read-only; do not write 'no anomaly' for items you haven't read.

Next step: Create your own initial draft of a post-mortem.

Next, use de-sensitized records approved by your company to run both prompts. Select a log, sort it chronologically, and separate it into facts and speculation columns; then take three items from the merged results and cross-check them against the original records. This video's narration is AI-generated. Thanks for watching.