侠Liangshen AIailiangshen.com
AI Implementation Workshop · Short topic lessons · Lesson 4

It says 'Done.' Do you believe it?

Ask for evidence, conduct spot checks, and verify externally yourself

Freeabout 10 minNo experience neededWeb lesson + slides + video

This is a skills module. We practice how to verify statements like 'Done,' 'Sent,' 'Calculated,' or 'Checked.' In ten minutes, you will learn to ask for evidence and perform spot checks.

By the end of this lesson, you can

  • Ask it to provide evidence: steps, details, and the actual number of items read
  • Select three items from the details: one from the beginning, one from the middle, one from the end, plus one outlier
  • Use a reconciliation sheet to separate 'what it said' from 'what you actually saw'
  • Write a verification checklist for automated tasks

Mnemonic: Don't trust words; check evidence. If evidence is insufficient, spot-check three lines. Verify externally yourself.

01

Opening

About 1 minutes

Let's look at a scenarioFictional example; numbers are illustrative

Project Manager He Lei

He asked Dragon Warrior to calculate the total person-days for three projects this month and confirm nothing was missed.

It replied: 'Calculated. Total is 126 person-days. Confirmed no omissions.'

He couldn't see how it was calculated, and he didn't know if 'confirmed' meant it was actually done or just said casually.

Such statements are easy to make and don't necessarily mean the work was done.
02

Method

About 4 minutes

Why does it say 'Done'?

It doesn't do the work itselfIn standard conversations, it cannot actually send messages, nor does it necessarily perform step-by-step calculations.
→
The outer layer of the shell.The application handles tool invocation, execution, and record-keeping on its behalf, then returns the results for it to evaluate.
→
The messages may contain errors.When it states 'task completed,' it is interpreting the message returned by the application. The message may not have been delivered, or it may have been misinterpreted.
Therefore, its statements must be verified against real-world records.

Request evidence based on the task type.

Calculation tasks

List the value and formula for each item, then provide the total.

Search tasks

List the relevant text excerpts and specify where they are located.

Review a batch of items

State how many items were actually read and how many were provided in total.

Tip

Prompt requesting evidence

Suggested time
Request evidence
You stated that the task is complete. Please provide evidence; do not repeat the conclusion.
1. List your steps and what each step accomplished.
2. If it is a calculation: list the value and formula for each item, then provide the total.
3. If it is a search: list the text excerpts you relied on and specify where they are located in the materials I provided.
4. If it is a review of a batch of items: state how many items you actually read and how many I provided in total.
5. Clearly distinguish: which items were actually processed, which were inferred, and which you were unable to execute (such as actually sending an email).
6. If any steps were not completed, state this directly; do not fill in the gaps.
It should explicitly state which actions it cannot perform, rather than fabricating a 'completed' status.
03

Practice

About 3 minutes

Two common types of false reports, in opposite directionsFictional examples

Claims completion, but was not completed

It states 'all checks completed, no anomalies,' but in reality, it only read a portion of the alerts.

Claims failure, but was actually completed

It states 'execution failed,' but the files in the folder were actually written correctly.

Items that were not read cannot be recorded as 'no anomalies.' First, verify the results in the real world.
Tip

Automated Task Acceptance Form

Suggested time
Generate Acceptance Form
Please help me design an 'Automated Task Acceptance Form' for the task: [Task Name].
List five items, each specifying: what to check, where to look, and what 'passing' looks like.
1. Is the output file present (correct name and date)?
2. Is the file structure complete (all required sections included)?
3. Sample two numbers and reconcile them with the original records.
4. Check the run logs: did it run, skip, or fail? If skipped or failed, what was the reason?
5. Does its self-report match the first four checks? If inconsistent, prioritize the first four checks.
Specify where to look for all five items, and clearly state in the fifth item that the first four checks take precedence.

Common Deviations and Remedies

5-Minute Exercise: Review a Fictional Run Log

Day 1

It says 'Completed,' with 120 alerts; the file has only one section, and the number is 102. Two discrepancies.

Day 2

It says 'Run Failed'; the file actually has all three sections, and the numbers match. Opposite direction.

Day 3

It says 'Completed'; no file for that day exists, and the log shows it was skipped due to computer sleep.

04

Quiz and Wrap-up

About 2 minutes

Three quick questions to check yourself

Question 1: It says 'Sent.' How should you verify?

Answer B.A statement in a conversation is not evidence; verify in the actual system logs.
Show the full transcript (video narration)
It says 'Done.' Do you believe it?

This is a skills module. We practice how to verify statements like 'Done,' 'Sent,' 'Calculated,' or 'Checked.' In ten minutes, you will learn to ask for evidence and perform spot checks.

After completing this module, you will be able to:

After completing this module, you will be able to do four things. First, ask it to provide evidence, including steps, details, and the actual number of items read. Second, select three items from the details: one from the beginning, one from the middle, one from the end, plus one outlier. Third, use a reconciliation sheet to separate 'what it said' from 'what you actually saw.' Fourth, write a verification checklist for automated tasks. The mnemonic is: Don't trust words; check evidence. If evidence is insufficient, spot-check three lines. Verify externally yourself.

Let's look at a scenario

Let's look at a fictional scenario; numbers are illustrative. Project Manager He Lei asked Dragon Warrior to calculate the total person-days for three projects this month and confirm nothing was missed. It replied: 'Calculated. Total is 126 person-days. Confirmed no omissions.' It sounds reassuring, but he couldn't see how it was calculated or whether 'confirmed' meant it was actually done. Such statements are easy to make and don't necessarily mean the work was done.

Why does it say 'Done'?

First, let's explain the mechanism. Its workflow operates in a loop: the Dragon Warrior determines the next step, and the outer layer of the shell—the application—invokes tools, executes tasks, saves records, and returns the results. Thus, when it says 'task completed,' it is actually reading the message returned by the application. The message may not have been delivered, or it may have been misread. In standard conversations, it cannot even send messages. Therefore, its statements must be verified against real-world records.

Request evidence based on the task type.

Request evidence based on the task type. For calculation tasks, list the value and formula for each item, then provide the total. For search tasks, list the relevant text excerpts and specify where they are located. For reviewing a batch of items, state how many were actually read and how many were provided in total. If these do not match, assume the check was not completed.

Prompt requesting evidence

This prompt requires it to list the steps, details, and the number of items read, clearly separating: which were actually processed, which were inferred, and which it was unable to execute. Particularly for actions like actually sending emails, which it typically cannot do in a conversation, it should directly state any uncompleted parts without filling them in.

Two common types of false reports, in opposite directions

Two types of false reports, in opposite directions. The first type: claims completion, but was not completed. It states 'all checks completed, no anomalies,' but in reality, it only read a portion of the alerts. Alerts that were not read cannot be recorded as 'no anomalies.' The second type: claims failure, but was actually completed. It states 'execution failed,' but the results in the folder were actually written correctly. Therefore, first verify the results in the real world by checking the files, rather than relying solely on what it says.

Automated Task Acceptance Form

After each run, review the output yourself and fill out the acceptance form. The five checks are: Is the file present? Is the structure complete? Do two sampled numbers reconcile? Do the logs show run, skip, or fail? Does its self-report match the first four checks? If inconsistent, prioritize the first four checks.

Common Deviations and Remedies

Several common deviations. It says 'Sent,' but verify in the actual sent folder. Sum of details doesn't match its total; ask it to recalculate item by item, with your manual calculation taking precedence. It says 'Failed,' but check the file first. One item missing from reconciliation: ask what rules cause it to skip records, update the process document, and don't just fix that one item. Three sampled items all pass, but this doesn't mean there are no issues; critical data should be reviewed by a second person at least once.

5-Minute Exercise: Review a Fictional Run Log

Use a fictional run log with illustrative numbers. A task that generates a daily duty alert summary runs for three consecutive days. Day 1: It says 'Completed,' 120 alerts, but the actual file has only one section, and the number is 102. Day 2: It says 'Run Failed,' but the actual file has all three sections. Day 3: It says 'Completed,' but there is no file for that day; the log shows it was skipped due to computer sleep. Please identify the two directions and write down the next step for each day.

Quiz

Here is a quiz. It says 'Sent.' How should you verify? A: Trust it; if it says sent, it's sent. B: Verify in the actual sent folder or system logs. C: Ask it again to confirm it hasn't changed its story. The answer is B. A statement in a conversation is not evidence; verify in the actual system logs.

Course Wrap-up

End of lesson. Copy a task you assigned it and the exact phrase you used to tell it to complete that task. Note a time when its report didn't match reality, and write down one sentence about it. Next time you see words like 'completed,' 'sent,' or 'confirmed,' pause and verify in the real world. Thanks for watching. This video's narration is AI-generated.