侠Liangshen AIailiangshen.com
AI Implementation Course · Core lessons · Lesson 9

Measuring Impact: Proving Savings with Data

Proving Savings with Data

Freeabout 60 minNo experience neededWeb lesson + slides + video

In previous lessons, you created a dedicated assistant and began testing it. This lesson answers one question: How much time did it actually save? Did quality drop?

By the end of this lesson, you can

  • Calculate time saved, quality changes, and costs using test-run data and the same acceptance checklist.
  • When samples are insufficient, supplement with historical records for back-testing, and present them separately.
  • Create an ROI table and write a one-page results report.
  • Make a judgment by comparing against pre-defined success and stop-loss thresholds.

Takeaways: A test-run data sheet, an ROI table, a draft one-page results report, and one judgment.

Video version: about 19 min

Narration is AI-generated. The video plays on YouTube; the chapter buttons above jump to the matching places. You can skip the video and scroll down to read.
01

Introduction and Concepts

About 4 minutes

Let's look at a scenarioFictional example; figures are illustrative

Yuanfan Technology · Administrative Specialist Lin Yue

She said: 'Using the dedicated assistant feels much faster.'

Department Manager Zheng asked: 'How much faster? Is it worth it?'

She couldn't answer. Feelings can't push resources; numbers can.

One person's positive feedback is an experience; a set of before-and-after data is evidence.

Three questions to answer when measuring effectiveness

Is it useful?Compared to the baseline using the same metrics, have time and quality truly improved?
→
Is it worth it?Include subscription costs, token usage, and review time to evaluate the return on investment.
→
Can it be scaled?Will the results remain consistent if a colleague takes over or you are on leave?
Record actuals as actuals and estimates as estimates. If no time was saved, state that honestly.
02

Baselines, Samples, and Fair Comparisons

About 7 minutes

Four 'same' criteria for fair comparison

The same tasks

Do not compare simple meetings with complex ones; record exceptionally simple or complex cases as usual, with notes explaining.

The same scope

The baseline and the trial run must cover the same range; include time spent reviewing and editing.

The same standards

Use the same acceptance checklist to judge quality; do not change the criteria for either the baseline or the trial run.

The same people

Performed by the same person, or by someone with equivalent skill level.

Do not cherry-pick good results: record failures, rework, and instances where the AI hindered progress.

How to establish a baseline: Re-evaluate past work using the same metrics

1
Use the acceptance checklist from Item 8 of the project card to score seven previous manually produced outputs.
2
Each violation counts as one missing item.
3
Review three outputs before the course, and complete the set to seven afterward.
4
The baseline must clearly state: the scope tested, the standards used, and which figures are estimates.
Time spent can be measured, but error counts must be measured using the same criteria as the trial run.

Lin Yue's Manual BaselineFictional example; figures are illustrative

Time per Session

90 minutes

Seven previous meeting records: from receiving the notes to issuing the minutes, including verification (actual measurement, illustrative)

Average Omissions

2.1 items

Seven records had a total of 15 omissions, averaging about 2.1 per session (actual measurement, illustrative)

These are the figures for manual note-taking; this is the baseline, not the assistant's performance.

What if the sample size is insufficient? Retrospective testing to supplement the sample.

1
Rule: If actual runs are fewer than 6, you must use retrospective testing on previous records to supplement the sample to more than 10.
2
Retrospective testing: Take de-identified real inputs from previous records and have the assistant generate drafts.
3
You check and time them using the real workflow, counting omissions with the same acceptance checklist.
4
In the 'Source' column: mark retrospective tests as 'Retrospective' and new tasks as 'Actual Run'.
In the report, write actual run and retrospective test figures separately, then write the combined numbers. Retrospective testing can only supplement the sample; it cannot replace actual runs.
03

Example: Four Metrics and the Digital Chain

About 13 minutes

Four metrics; each must clearly state how it is measured.

Time

Total minutes from start to delivery, including checking and editing; use a phone timer.

Quality and Errors

Use the same checklist to count the final omissions before submission.

Consistency

Number of times all formats met standards ÷ Total number of times; this can only be promoted when colleagues take over or you are on leave.

Cost

Subscription fees, token fees, and one-time investment in hours; amounts must come from your own bills.

Faster but with more errors does not count as a win.

The ins and outs of cost: Where tokens come from

1
What you send in is the entire context: the script, the dossier, all previous questions and answers, plus this new sentence.
2
The output is measured separately, usually at a higher unit price, based on your current version.
3
Cost-saving method 1: Start a new conversation for each task.
4
Cost-saving method 2: If a conversation gets too long, ask it to write a summary first, then start a new conversation with that summary.
The script and dossier should be concise, but the guardrail clauses must not be deleted just to save costs.

Ten identical questions, two different conversation approachesUnit shown is 'portions', not actual token counts

Asking ten rounds in a single conversation

200 portions

Round 1: 11 portions; Round 10: 29 portions; Total for ten rounds (illustrative)

Starting a new conversation for each of ten tasks

110 portions

Each conversation asks only one round, 11 portions per round; Total for ten rounds (illustrative)

This assumes the ten questions are independent; if they depend on each other, keep them in the same conversation.

Lin Yue's data chain: Three sets of numbers must not be mixed

Manual baselineReviewing 7 past cases with the same checklist: approx. 90 minutes, 2.1 missed items
→
Assistant's first trialSame 7 cases retested: approx. 35 minutes, 1.3 missed items
→
After adjusting the scriptRetesting 7 cases plus 4 live runs, totaling 11 cases: approx. 28 minutes, 0.4 missed items
The 2.1 missed items refer to manual meeting minutes; the count dropped to 0.4 only after adjusting the script.

Separate the combined numbers to see the truthFictional example; figures are illustrative

Retesting 7 cases

Approx. 30 minutes

Total missed items: 3, averaging 0.4 per run. This is based on the initial 7 trial runs, re-executed after script adjustments, so the results are somewhat optimistic.

Actual runs: 4

Approx. 25 minutes

Total missed items: 1, averaging 0.25 per run. Running once a week yields the cleanest and most reliable results, but only covers 4 runs.

Combined 11 runs: approx. 28 minutes, with about 0.4 missed items. Combined figures should be reported, but separate figures are more honest.

Industry Edition: Qilan Intelligence's Work Order Classification

1
Daily Time SpentDropped from approx. 90 minutes to approx. 45 minutes, including review (actual test, illustrative)
2
First Response TimeDropped from approx. 3 hours to approx. 1.5 hours, derived from system timestamps (illustrative)
3
Misrouting RateBaseline is approx. 10%; trial sampling of 20 tickets resulted in 1 error. The sample size is too small, so the conclusion is tentative.
An error rate of 1 in 20 cannot be stated as 'misrouting rate stably dropped to 5%.'
04

Practice A: Organize Data, Colleague Spot-Check

About 6 minutes
Tip

Exercise A: Organize Trial Run Data

Suggested time Approx. 6 minutes
Prompt A
You are a meticulous data organizer. Below are my trial run records, with each row representing one run: Serial Number, Source (Backtest or Actual Run), Total Time (minutes, including review), Final Missed Items Count, and Whether Manual Correction is Needed.
[Paste your table here]

Please:
1. Repeat the data I provided row by row to confirm no errors in reading.
2. Calculate the average, maximum, and minimum time, average missed items, and count of manual corrections needed for backtests, actual runs, and combined runs separately.
3. Additionally, calculate the average time and average missed items for the 'last 4 runs' (i.e., actual runs). I need both the overall average and the stable values written out.
4. Show the calculation process for each number so I can verify it myself.
5. Do not guess data I did not provide; directly point out where data is missing.
The data it repeats must match the original table; missing data should be pointed out, not fabricated; averages must be verified by you.

Colleague Spot-Check: Don't Just Trust Your Own Missed Item Counts

Lin Yue draws 3 samples: She counts 1 item, her colleague counts 2 items, resulting in a difference of 1 item. After standardizing the criteria, they recount (as an illustration). If the difference exceeds 1 item, identify the cause first, then draw conclusions.
05

Practice B: Calculate Costs

About 12 minutes

Input-output table: Calculate cell by cell

Time saved per instanceBaseline minus trial average: 90 - 28 = 62 minutes
→
Time saved per week62 minutes × 1 instance ÷ 60, approximately 1.03 hours
→
Time saved per monthBased on four weeks, approximately 4.1 hours
→
Monthly net benefitMonetary version: 4.1 hours × 100 RMB - 150 RMB = 260 RMB
Label each cell as either 'actual measurement' or 'estimate'. All monetary figures must come from your own bills, not from AI-generated data.

Payback period: Monetary version vs. Hourly versionIllustrative amounts, not real prices

Monetary version

Approximately 9.2 weeks

600 RMB ÷ weekly net benefit of 65 RMB (i.e., monthly net benefit of 260 RMB ÷ 4)

Hourly version

Approximately 5.8 weeks

6 ÷ 1.03: Subscription costs are not included

Do not mix the two versions. Clearly state which version is used in your report.
Tip

Exercise B: Have it check only the formulas

Suggested time Approximately 4 minutes
Prompt B
I am creating an input-output table. Below are my data and formulas. Please perform only one task: check if the formulas are correct, if the units match, and if there are any duplicate calculations. Do not modify my numbers, and do not fabricate prices for me.
Minutes saved per instance = [Baseline time per instance] - [Trial average time] = [Result]
Instances per week = [Number]
Monthly subscription fee = [My bill amount]
Monthly token cost = [My usage amount, or write 'included in subscription']
Hourly value of time = [My standard]
One-time investment in hours = [Number]
Please list any issues found item by item; if none, write 'No issues found'.
It checks only formulas and units, does not modify your numbers, and does not verify prices for you.

Input-Output Table: Verification Criteria

There are three ways to define the hourly value of time. Choose one and specify: convert from monthly salary, use internal company labor cost, or simply state the hours without converting to money.
06

Practice C: Write a One-Page Results Report

About 9 minutes

One-Page Results Report: Six Sections

1
One Problem: What the task was like before, and where it got stuck
2
One Method: What process was used, and how humans and the Dragon Warrior divided the work
3
One Set of Data: Before-and-after comparison of four metrics, specifying sample size, with live runs and backtests separated
4
What Colleagues Said: Two to three direct quotes
5
Unexpected Findings: One thing that differed from your expectations
6
Conclusion and Next Steps: Is it worth it? (One sentence); What resources, scope, and timeline are requested
Use only the numbers and direct quotes you provided. If no savings were achieved, state this truthfully and explain why.

How to Get 'What Colleagues Said': The Five Questions for ColleaguesThe Five Questions for Colleagues: You watch from the side, without intervening or teaching.

Recording Rules

Ask a colleague to try it, then ask five questions after they finish.

1. What did you use it for? 2. At what moment did you feel 'It understood my work'? 3. At what moment did you feel it was talking nonsense? 4. Will you use it in the future? 5. Describe it in one word.

Record direct quotes. If they say 'It was okay,' you write 'It was okay.'

Direct quotes are more valuable than your summary. The AI can help you polish the text, but it cannot fabricate a colleague's genuine reaction.
Tip

Exercise C: Have it write the report in six sections

Suggested time Approx. 9 minutes
Prompt C
You are a senior assistant specializing in writing work reports. Based on the data I provide, write a one-page results report strictly following these six sections: 1. Problem; 2. Actions Taken; 3. Data; 4. Colleague Feedback; 5. Unexpected Findings; 6. Conclusion and Next Steps.
Requirements: Use only the numbers I provide. Keep the labels 'actual' or 'estimated' next to each key number. Quote colleague comments exactly as provided; do not rewrite, merge, or embellish them. If the sample size is small, explicitly state 'Sample size is small; conclusions are tentative.' Do not add any data or effects I did not provide. Maintain a plain tone and avoid exaggeration. Keep the entire text under 500 words.
My materials:
Project Name: [Name]
Baseline Situation: [Baseline, including time per instance, weekly frequency, and number of missed items, noting 'actual' or 'estimated']
My Actions: [Process and human-AI division of labor, in two or three sentences]
Trial Run Results: [Number of actual runs and backtests; respective average times and missed items; combined numbers; stable values from later runs]
Spot Checks: [Number of documents checked by colleagues and the degree of discrepancy]
Costs: [Subscription fees, token costs, and one-time investment in hours, noting 'estimated']
Colleague Quotes: [2 to 3 sentences, exact quotes]
Unexpected Findings: [One thing that differed from expectations]
Requested Next Steps: [Resources, scope, and timeline]
It cannot add effects you did not provide. Remove all numbers and descriptions that were not provided.

Checklist for a One-Page Report

If no one is willing to try it, honestly state why.
07

Judgment, Quizzes, and Assignments

About 9 minutes

Judgment: Compare against the pre-written benchmarks.Lin Yue's judgment, numbers are illustrative.

Success Benchmark

  • Average time does not exceed half the baseline (45 minutes).
  • Average missed items do not exceed 1.0 and are lower than the baseline of 2.1.
  • No red-line incidents.

Stop-Loss Benchmark

  • One red-line incident occurs.
  • Actual run time for two consecutive instances is not less than 90 minutes.
  • After 4 cumulative instances, the average time still exceeds 72 minutes.
Lin Yue's results: 28 minutes, 0.4 missed items, 0 incidents. Judgment: Success achieved. Proceed.

Three quick questions to check yourself

Question 1: There are only 4 actual runs; the sample is insufficient. What should you do?

Answer B.Backtesting only supplements samples; it cannot replace live runs. The two parts must be documented separately.

Question 2: What does Lin Yue mean by 'Item 2.1'?

Answer B.The baseline, the first trial, and the post-adjustment figures are three distinct numbers and must not be confused.

Question 3: A colleague's spot check reveals a discrepancy of three items between the total of three instances and your count. What should you do?

Answer B.A discrepancy of more than one item indicates that the acceptance checklist criteria are not yet consistent.

Assignment and Project Card Updates

Project Card: Column 10 is closed, Column 11 contains the results, and Column 12 contains the results report draft. Optional: Compare different gears using the same script.
Show the full transcript (video narration)
Measuring Impact

Welcome to Lesson 9 of the 'AI Implementation Course,' focusing on measuring impact. In previous lessons, you created a dedicated assistant and began testing it. This lesson answers one question: How much time did it actually save? Did quality drop? Is it worth it? We use data to prove it, not gut feelings.

By the end of this lesson, you will be able to

By the end of this lesson, you will be able to do four things. First, calculate time saved, quality changes, and costs using test-run data and the same acceptance checklist. Second, when samples are insufficient, supplement with historical records for back-testing, and present them separately. Third, create an ROI table and write a one-page results report. Fourth, make a judgment by comparing against pre-defined success and stop-loss thresholds.

Let's look at a scenario

First, consider a scenario that is fictional, with numbers used only for illustration. Lin Yue, an administrative specialist at Yuanfan Technology, used the dedicated assistant and told her department manager, Mr. Zheng, 'It feels much faster.' Mr. Zheng then asked, 'How much faster? Is it worth it?' Lin Yue couldn't answer immediately. Feelings can't drive resource allocation; numbers can.

Three questions to answer when measuring effectiveness

Measuring effectiveness requires answering three questions: Is it useful? Is it worth it? Can it be scaled? First, compare it side-by-side with the baseline to see if time and quality have genuinely improved. Second, factor in subscription fees, token costs, and the time you spend reviewing to assess the return on investment. Third, ensure results remain consistent even if a colleague takes over or you are on leave. Be honest with every number: record actuals as actuals and estimates as estimates. If no time was saved, state that clearly.

Four 'same' criteria for fair comparison

When making comparisons, adhere to the four 'same' criteria. The same tasks: Do not compare simple meetings with complex ones. The same scope: The baseline and the trial run must cover the same range; include three minutes for AI drafting and twenty minutes for your review and editing. The same standards: Use the same acceptance checklist; do not change criteria for either the baseline or the trial run. The same people: Performed by the same person, or by someone with equivalent skill level. Additionally, do not cherry-pick good results. Record failures, rework, and instances where the AI hindered progress.

How to establish a baseline: Re-evaluate past work using the same metrics

How to establish a baseline. Many people’s baselines only quantify time, while error counts are estimated by impression. This step requires using the same measuring stick as the trial run: take the acceptance checklist from Column 8 of the project card and retrospectively score seven previous manual outputs. For each violation, count one omission. Review three samples before the course and complete the remaining four after the course. The baseline must also clearly state three things: the scope being measured, the standards used, and which figures are estimates.

Lin Yue's Manual Baseline

Looking at Lin Yue's manual baseline: this is fictional, and the numbers are illustrative. The seven previous meeting records were retrospectively scored using the same seven-item acceptance checklist. The average time per session was 90 minutes, an illustrative actual measurement. The seven records had a total of 15 omissions, averaging about 2.1 per session. These are the figures for manual note-taking; this is the baseline, not the assistant's performance, so they must not be mixed when making comparisons later.

What if the sample size is insufficient? Retrospective testing to supplement the sample.

What if the actual run sample is insufficient? There is only one rule: if actual runs are fewer than six, you must use retrospective testing on previous records to supplement the sample to more than ten. The retrospective testing process is: take de-identified real inputs from previous records and have the assistant generate drafts; you check and time them using the real workflow, counting omissions with the same acceptance checklist; mark whether each source is 'Retrospective' or 'Actual Run' in the Source column. In the report, write the actual run and retrospective test figures separately, then write the combined numbers. Note that retrospective tests tend to be optimistic because you know what the record should contain. Therefore, retrospective testing can only supplement the sample; it cannot replace actual runs.

Four metrics; each must clearly state how it is measured.

Looking only at how much faster it is is not enough. Faster but with more errors does not count as a win. Four metrics; each must clearly state how it is measured. Time: Total minutes from start to delivery, including your time for checking and editing; use a phone timer, do not fill in estimates afterward. Quality and Errors: Use the same acceptance checklist to count the final omissions before submission. Consistency: The number of times all formats met standards, divided by the total number of times. Cost: Subscription fees, token fees, and one-time investment in hours; amounts must come from your own bills.

The ins and outs of cost: Where tokens come from

Many people only see the total token cost without knowing where it comes from. Here are three key points. First, what you send in each time is not just your last sentence, but the entire context: the script, the dossier, all previous questions and answers, plus this new sentence. The entire stack is resent in every round, so the longer the conversation, the more is sent each time. Second, the output is measured separately, usually at a higher unit price, based on your current version. Third, here are two cost-saving methods: start a new conversation for each task; if a conversation gets too long, ask it to write a summary first, then start a new conversation with that summary. The script and dossier should be concise, but the guardrail clauses must not be deleted just to save costs. The number of tokens per Chinese character varies by provider, so do not copy English conversion rates, and do not calculate costs using ratios provided by others.

Ten identical questions, two different conversation approaches

Use an illustrative table to see the trend. Assume the script and dossier total 10 portions, with each question and answer counting as 1 portion, counting only the input. In a single conversation with ten rounds, Round 1 is 11 portions, and by Round 10 it reaches 29 portions, totaling 200 portions for ten rounds. Starting a new conversation for each of ten tasks, with 11 portions per round, totals 110 portions. The total input for a long conversation is nearly double that of starting new conversations, and it increases as the conversation progresses. These are illustrative units, not actual token counts. This assumes the ten questions are independent; if they depend on each other, keep them in the same conversation.

Lin Yue's data chain: Three sets of numbers must not be mixed

These are numbers for the same person, using the same acceptance checklist, across three stages. The first is the manual baseline: reviewing 7 past cases with the same checklist, averaging about 90 minutes, with 2.1 missed items. The second is the assistant's first trial: retesting the same 7 cases, taking about 35 minutes, with 1.3 missed items. The third is after adjusting the script: retesting 7 cases plus 4 live runs over four weeks, totaling 11 cases, taking about 28 minutes, with 0.4 missed items. Note that the 2.1 missed items refer to manual meeting minutes, not the assistant's performance. These three sets of numbers cannot be mixed.

Separate the combined numbers to see the truth

Break down the combined figures. Reviewing seven runs averages about 30 minutes, with 3 total missed items, which is somewhat optimistic because the script was adjusted before re-running. Four actual runs average about 25 minutes, with 1 total missed item (0.25 per run). Actual runs are the cleanest and most reliable, but limited to four. Combining 11 runs averages about 28 minutes, with about 0.4 missed items. The learning curve should also be documented: early runs are typically slower. Reports should include both the overall average and the stable values from later runs, not just the latter.

Industry Edition: Qilan Intelligence's Work Order Classification

Technology Industry Edition. Qilan Intelligence's Luo Xiaotang performs these tasks daily, providing a sufficient sample size without needing backtesting. The figures here are illustrative. Daily time spent dropped from approx. 90 minutes to approx. 45 minutes, including review. First response time dropped from approx. 3 hours to approx. 1.5 hours, derived from system timestamps. Misrouting rate: baseline approx. 10%; trial sampling of 20 tickets resulted in 1 error. The sample size is too small; do not state it stably dropped to 5%. Clearly state the conclusion is tentative, and plan to sample 40 tickets in the next round.

Exercise A: Organize Trial Run Data

Exercise A: Organize trial run data and verify using a calculator. Organize your trial run records into a table, separating actual runs from backtests, and calculate average time, average missed items, and consistency rate. If you lack your own data, use Lin Yue's 11-row table. Use this prompt to have it act as a data organizer. It should first repeat your data to confirm accuracy, then calculate figures for backtests, actual runs, and combined runs separately, and show the calculation process for your verification. It should directly point out missing data, not fabricate it. You must calculate the averages yourself.

Colleague Spot-Check: Don't Just Trust Your Own Missed Item Counts

You are most likely to miss items when checking your own work. Have a colleague randomly select three outputs, without looking at your numbers, and independently count them using the same checklist. Then compare the results. When Lin Yue conducted a spot check, she counted one item, while her colleague counted two. The reason was that the colleague considered a deadline listed only as 'Friday' without a specific date as a missing item, whereas Lin Yue had considered it compliant. They then updated the first item on the checklist to require a specific date and recounted according to the new standard. The rule is simple: if the total difference across three samples exceeds one item, identify the cause and standardize the criteria before drawing conclusions. Self-check against the inspection standards below.

Input-output table: Calculate cell by cell

To create an input-output table, calculate cell by cell. The baseline minus the trial average gives Lin Yue's figure: 90 minus 28, saving 62 minutes per instance. With one instance per week, the weekly savings are approximately 1.03 hours. Based on four weeks, the monthly savings are approximately 4.1 hours, which is a conservative estimate. In the monetary version, the hourly value of time is set at 100 RMB. The monthly value of saved time is 410 RMB. Subtracting the monthly subscription fee of 150 RMB, the monthly net benefit is 260 RMB. These amounts are illustrative and not real prices. Label each cell as either 'actual measurement' or 'estimate'.

Payback period: Monetary version vs. Hourly version

Payback period has two versions. Monetary version: A one-time investment of 6 hours, valued at 100 RMB per hour, totaling 600 RMB, divided by the weekly net benefit of 65 RMB, resulting in approximately 9.2 weeks. The weekly net benefit is the monthly net benefit of 260 RMB divided by 4. Hourly version: The one-time investment in hours divided by the weekly saved hours, 6 divided by 1.03, resulting in approximately 5.8 weeks. This is faster than the monetary version because it does not include subscription costs. Do not mix the two versions; clearly state which version is used in your report.

Exercise B: Have it check only the formulas

Exercise B: Do the Math. Let it do only one thing: check the formula. You fill in the numbers; it does not change them or make up prices for you. It checks whether the formula is correct, whether the units match, and whether any costs are double-counted. There are three common errors at this stage: asking the AI to look up prices, which may be outdated or incorrect; calculating only subscriptions and missing the one-time setup time; or deducting the review time twice.

Input-Output Table: Verification Criteria

Verify the Input-Output Table against these criteria. Every cell must contain a number or clearly state 'included in subscription,' with no blanks. Every number must be labeled as measured or estimated. In the cost section, ensure review time is already included in the hours used, so do not deduct it again. The payback period must be in weeks, not months, and specify whether it is the monetary or hourly version. All monetary figures must come from your own billing records. Additionally, there are three common ways to define the hourly value of time: converting from monthly salary, using internal company labor cost, or simply stating the hours without converting to money. Choose one and specify clearly.

One-Page Results Report: Six Sections

Exercise C: Write a One-Page Results Report. Structure: six sections, one or two sentences each. Section 1, One Problem: What the task was like before, and where it got stuck. Section 2, One Method: What process was used, and how humans and the Dragon Warrior divided the work. Section 3, One Set of Data: Before-and-after comparison of four metrics, specifying sample size, with live runs and backtests separated. Section 4, What Colleagues Said: Two to three direct quotes. Section 5, Unexpected Findings: One thing that differed from your expectations. Section 6, Conclusion and Next Steps: Is it worth it? (One sentence); What resources, scope, and timeline are requested. If no savings were achieved, state this truthfully and explain why.

How to Get 'What Colleagues Said': The Five Questions for Colleagues

Where does 'What Colleagues Said' in the report come from? Ask a colleague to try it, while you watch from the side without intervening or teaching. After they finish, ask them five questions: What did you use it for? At what moment did you feel it understood your work? At what moment did you feel it was talking nonsense? Will you use it in the future? Describe it in one word. Record direct quotes; do not summarize for them or beautify the language. If they say 'It was okay,' you write 'It was okay.' Direct quotes are more valuable than your summary. The AI can help you polish the report, but it cannot fabricate a colleague's genuine reaction. If you do not yet have colleague quotes before class, you can practice the structure with sample data and replace it with your own after class.

Exercise C: Have it write the report in six sections

Use this prompt to have it write a report in six sections. Its requirements are: use only the numbers you provide, keeping 'actual' or 'estimated' next to each key number; quote colleague comments exactly without rewriting, merging, or embellishing; state 'conclusions are tentative' when sample sizes are small; keep the text under 500 words. Note that it sometimes adds flattering phrases like 'significant efficiency improvement' on its own. Remove all numbers and descriptions that were not provided.

Checklist for a One-Page Report

Checklist for a One-Page Report, reviewed item by item. One page, six complete sections, every number traceable to the log, with 'actual' or 'estimated' labels. Write actual runs and backtests separately. Colleague comments must be exact quotes, not your summaries. Unexpected findings must not be written as 'consistent with expectations'; if there truly are none, it means observation was insufficient, so review the records again. The conclusion should be one sentence; next steps must include scope and timeline. If no one is willing to try it, honestly state why.

Judgment: Compare against the pre-written benchmarks.

Judgment: Compare against the pre-written success and stop-loss benchmarks. These benchmarks must be defined before trial runs and cannot be changed afterward. Lin Yue's success benchmarks are threefold: average time not exceeding half the baseline (45 minutes); average missed items not exceeding 1.0 and lower than the baseline of 2.1; no red-line incidents. The stop-loss benchmarks are also threefold; triggering any requires writing four sentences honestly. Lin Yue's results: average 28 minutes, 0.4 missed items, 0 red-line incidents. None of the stop-loss benchmarks were triggered. Judgment: Success achieved. Proceed and request small-scale rollout. The judgment has three tiers: Achieved, Partially Achieved, Not Achieved.

Quiz Question 1

Question 1: Live runs only produced four instances, which is insufficient. What should you do? A. Draw conclusions directly. B. Use historical backtesting to supplement the sample to over ten items, and present them separately in the report. C. Copy the four instances to ten. The answer is B. Backtesting only supplements samples; it cannot replace live runs. The two parts must be documented separately.

Quiz Question 2

Question 2: What does Lin Yue mean by 'Item 2.1'? A. The missing item from the assistant's first trial. B. The missing item from manual meeting minutes, i.e., the manual baseline. C. The missing item after adjusting the script. The answer is B. The baseline, the first trial, and the post-adjustment figures are three distinct numbers and must not be confused.

Quiz Question 3

Question 3: A colleague's spot check reveals a discrepancy of three items between the total of three instances and your count. What should you do? A. Trust your own count. B. First identify the cause and standardize the criteria, then draw conclusions. C. Trust the colleague's count and stop explaining. The answer is B. A discrepancy of more than one item indicates that the acceptance checklist criteria are not yet consistent.

Assignment and Project Card Updates

Assignment: The following six items are mandatory. Organized trial records: Live runs and backtesting are separated. The total must be at least 10 items; if fewer, record the actual number. Baseline records evaluated using the same acceptance checklist. Colleague spot-check records. Input-output table, with each cell marked as actual measurement or estimate. One-page results report draft, including judgment. Original quotes from at least one colleague's five-question interview. Project Card: Column 10 is closed, Column 11 contains the results, and Column 12 contains the results report draft. The AI can help polish the text, but it cannot fabricate your time logs or your colleagues' original quotes.

Next Lesson: Bringing it Back to the Company

The core of this lesson is to clarify the numbers. Remember three things: First, comparisons must be fair—same tasks, same standards, same people. Second, separate live runs from backtests; if the sample size is insufficient, state the conclusion as tentative. Third, if no savings were achieved, state that fact honestly. In the next lesson, take it back to the company: use a three-minute report to present it to your boss, and then have colleagues try it out for real. Please bring today's results report draft and judgment. Thank you for watching. See you in the next lesson. This video's narration is AI-generated.