Video version: about 19 min
Introduction and Concepts
About 4 minutesLet's look at a scenarioFictional example; figures are illustrative
She said: 'Using the dedicated assistant feels much faster.'
Department Manager Zheng asked: 'How much faster? Is it worth it?'
She couldn't answer. Feelings can't push resources; numbers can.
Three questions to answer when measuring effectiveness
Baselines, Samples, and Fair Comparisons
About 7 minutesFour 'same' criteria for fair comparison
The same tasks
Do not compare simple meetings with complex ones; record exceptionally simple or complex cases as usual, with notes explaining.
The same scope
The baseline and the trial run must cover the same range; include time spent reviewing and editing.
The same standards
Use the same acceptance checklist to judge quality; do not change the criteria for either the baseline or the trial run.
The same people
Performed by the same person, or by someone with equivalent skill level.
How to establish a baseline: Re-evaluate past work using the same metrics
Lin Yue's Manual BaselineFictional example; figures are illustrative
Time per Session
Seven previous meeting records: from receiving the notes to issuing the minutes, including verification (actual measurement, illustrative)
Average Omissions
Seven records had a total of 15 omissions, averaging about 2.1 per session (actual measurement, illustrative)
What if the sample size is insufficient? Retrospective testing to supplement the sample.
Example: Four Metrics and the Digital Chain
About 13 minutesFour metrics; each must clearly state how it is measured.
Time
Total minutes from start to delivery, including checking and editing; use a phone timer.
Quality and Errors
Use the same checklist to count the final omissions before submission.
Consistency
Number of times all formats met standards ÷ Total number of times; this can only be promoted when colleagues take over or you are on leave.
Cost
Subscription fees, token fees, and one-time investment in hours; amounts must come from your own bills.
The ins and outs of cost: Where tokens come from
Ten identical questions, two different conversation approachesUnit shown is 'portions', not actual token counts
Asking ten rounds in a single conversation
Round 1: 11 portions; Round 10: 29 portions; Total for ten rounds (illustrative)
Starting a new conversation for each of ten tasks
Each conversation asks only one round, 11 portions per round; Total for ten rounds (illustrative)
Lin Yue's data chain: Three sets of numbers must not be mixed
Separate the combined numbers to see the truthFictional example; figures are illustrative
Retesting 7 cases
Total missed items: 3, averaging 0.4 per run. This is based on the initial 7 trial runs, re-executed after script adjustments, so the results are somewhat optimistic.
Actual runs: 4
Total missed items: 1, averaging 0.25 per run. Running once a week yields the cleanest and most reliable results, but only covers 4 runs.
Industry Edition: Qilan Intelligence's Work Order Classification
Practice A: Organize Data, Colleague Spot-Check
About 6 minutesExercise A: Organize Trial Run Data
Suggested time Approx. 6 minutesYou are a meticulous data organizer. Below are my trial run records, with each row representing one run: Serial Number, Source (Backtest or Actual Run), Total Time (minutes, including review), Final Missed Items Count, and Whether Manual Correction is Needed. [Paste your table here] Please: 1. Repeat the data I provided row by row to confirm no errors in reading. 2. Calculate the average, maximum, and minimum time, average missed items, and count of manual corrections needed for backtests, actual runs, and combined runs separately. 3. Additionally, calculate the average time and average missed items for the 'last 4 runs' (i.e., actual runs). I need both the overall average and the stable values written out. 4. Show the calculation process for each number so I can verify it myself. 5. Do not guess data I did not provide; directly point out where data is missing.
Colleague Spot-Check: Don't Just Trust Your Own Missed Item Counts
Practice B: Calculate Costs
About 12 minutesInput-output table: Calculate cell by cell
Payback period: Monetary version vs. Hourly versionIllustrative amounts, not real prices
Monetary version
600 RMB ÷ weekly net benefit of 65 RMB (i.e., monthly net benefit of 260 RMB ÷ 4)
Hourly version
6 ÷ 1.03: Subscription costs are not included
Exercise B: Have it check only the formulas
Suggested time Approximately 4 minutesI am creating an input-output table. Below are my data and formulas. Please perform only one task: check if the formulas are correct, if the units match, and if there are any duplicate calculations. Do not modify my numbers, and do not fabricate prices for me. Minutes saved per instance = [Baseline time per instance] - [Trial average time] = [Result] Instances per week = [Number] Monthly subscription fee = [My bill amount] Monthly token cost = [My usage amount, or write 'included in subscription'] Hourly value of time = [My standard] One-time investment in hours = [Number] Please list any issues found item by item; if none, write 'No issues found'.
Input-Output Table: Verification Criteria
Practice C: Write a One-Page Results Report
About 9 minutesOne-Page Results Report: Six Sections
How to Get 'What Colleagues Said': The Five Questions for ColleaguesThe Five Questions for Colleagues: You watch from the side, without intervening or teaching.
Ask a colleague to try it, then ask five questions after they finish.
1. What did you use it for? 2. At what moment did you feel 'It understood my work'? 3. At what moment did you feel it was talking nonsense? 4. Will you use it in the future? 5. Describe it in one word.
Record direct quotes. If they say 'It was okay,' you write 'It was okay.'
Exercise C: Have it write the report in six sections
Suggested time Approx. 9 minutesYou are a senior assistant specializing in writing work reports. Based on the data I provide, write a one-page results report strictly following these six sections: 1. Problem; 2. Actions Taken; 3. Data; 4. Colleague Feedback; 5. Unexpected Findings; 6. Conclusion and Next Steps. Requirements: Use only the numbers I provide. Keep the labels 'actual' or 'estimated' next to each key number. Quote colleague comments exactly as provided; do not rewrite, merge, or embellish them. If the sample size is small, explicitly state 'Sample size is small; conclusions are tentative.' Do not add any data or effects I did not provide. Maintain a plain tone and avoid exaggeration. Keep the entire text under 500 words. My materials: Project Name: [Name] Baseline Situation: [Baseline, including time per instance, weekly frequency, and number of missed items, noting 'actual' or 'estimated'] My Actions: [Process and human-AI division of labor, in two or three sentences] Trial Run Results: [Number of actual runs and backtests; respective average times and missed items; combined numbers; stable values from later runs] Spot Checks: [Number of documents checked by colleagues and the degree of discrepancy] Costs: [Subscription fees, token costs, and one-time investment in hours, noting 'estimated'] Colleague Quotes: [2 to 3 sentences, exact quotes] Unexpected Findings: [One thing that differed from expectations] Requested Next Steps: [Resources, scope, and timeline]
Checklist for a One-Page Report
Judgment, Quizzes, and Assignments
About 9 minutesJudgment: Compare against the pre-written benchmarks.Lin Yue's judgment, numbers are illustrative.
Success Benchmark
- Average time does not exceed half the baseline (45 minutes).
- Average missed items do not exceed 1.0 and are lower than the baseline of 2.1.
- No red-line incidents.
Stop-Loss Benchmark
- One red-line incident occurs.
- Actual run time for two consecutive instances is not less than 90 minutes.
- After 4 cumulative instances, the average time still exceeds 72 minutes.
Three quick questions to check yourself
Question 1: There are only 4 actual runs; the sample is insufficient. What should you do?
Question 2: What does Lin Yue mean by 'Item 2.1'?
Question 3: A colleague's spot check reveals a discrepancy of three items between the total of three instances and your count. What should you do?
Assignment and Project Card Updates
Next Lesson: Bringing it Back to the Company
Three-minute presentation, colleague trial, governance, and review
- Bring the previous page's results report draft and judgment.
- Next lesson: Use it for a three-minute report, and rehearse the boss's follow-up questions.
Show the full transcript (video narration)
Measuring Impact
Welcome to Lesson 9 of the 'AI Implementation Course,' focusing on measuring impact. In previous lessons, you created a dedicated assistant and began testing it. This lesson answers one question: How much time did it actually save? Did quality drop? Is it worth it? We use data to prove it, not gut feelings.
By the end of this lesson, you will be able to
By the end of this lesson, you will be able to do four things. First, calculate time saved, quality changes, and costs using test-run data and the same acceptance checklist. Second, when samples are insufficient, supplement with historical records for back-testing, and present them separately. Third, create an ROI table and write a one-page results report. Fourth, make a judgment by comparing against pre-defined success and stop-loss thresholds.
Let's look at a scenario
First, consider a scenario that is fictional, with numbers used only for illustration. Lin Yue, an administrative specialist at Yuanfan Technology, used the dedicated assistant and told her department manager, Mr. Zheng, 'It feels much faster.' Mr. Zheng then asked, 'How much faster? Is it worth it?' Lin Yue couldn't answer immediately. Feelings can't drive resource allocation; numbers can.
Three questions to answer when measuring effectiveness
Measuring effectiveness requires answering three questions: Is it useful? Is it worth it? Can it be scaled? First, compare it side-by-side with the baseline to see if time and quality have genuinely improved. Second, factor in subscription fees, token costs, and the time you spend reviewing to assess the return on investment. Third, ensure results remain consistent even if a colleague takes over or you are on leave. Be honest with every number: record actuals as actuals and estimates as estimates. If no time was saved, state that clearly.
Four 'same' criteria for fair comparison
When making comparisons, adhere to the four 'same' criteria. The same tasks: Do not compare simple meetings with complex ones. The same scope: The baseline and the trial run must cover the same range; include three minutes for AI drafting and twenty minutes for your review and editing. The same standards: Use the same acceptance checklist; do not change criteria for either the baseline or the trial run. The same people: Performed by the same person, or by someone with equivalent skill level. Additionally, do not cherry-pick good results. Record failures, rework, and instances where the AI hindered progress.
How to establish a baseline: Re-evaluate past work using the same metrics
How to establish a baseline. Many people’s baselines only quantify time, while error counts are estimated by impression. This step requires using the same measuring stick as the trial run: take the acceptance checklist from Column 8 of the project card and retrospectively score seven previous manual outputs. For each violation, count one omission. Review three samples before the course and complete the remaining four after the course. The baseline must also clearly state three things: the scope being measured, the standards used, and which figures are estimates.
Lin Yue's Manual Baseline
Looking at Lin Yue's manual baseline: this is fictional, and the numbers are illustrative. The seven previous meeting records were retrospectively scored using the same seven-item acceptance checklist. The average time per session was 90 minutes, an illustrative actual measurement. The seven records had a total of 15 omissions, averaging about 2.1 per session. These are the figures for manual note-taking; this is the baseline, not the assistant's performance, so they must not be mixed when making comparisons later.
What if the sample size is insufficient? Retrospective testing to supplement the sample.
What if the actual run sample is insufficient? There is only one rule: if actual runs are fewer than six, you must use retrospective testing on previous records to supplement the sample to more than ten. The retrospective testing process is: take de-identified real inputs from previous records and have the assistant generate drafts; you check and time them using the real workflow, counting omissions with the same acceptance checklist; mark whether each source is 'Retrospective' or 'Actual Run' in the Source column. In the report, write the actual run and retrospective test figures separately, then write the combined numbers. Note that retrospective tests tend to be optimistic because you know what the record should contain. Therefore, retrospective testing can only supplement the sample; it cannot replace actual runs.
Four metrics; each must clearly state how it is measured.
Looking only at how much faster it is is not enough. Faster but with more errors does not count as a win. Four metrics; each must clearly state how it is measured. Time: Total minutes from start to delivery, including your time for checking and editing; use a phone timer, do not fill in estimates afterward. Quality and Errors: Use the same acceptance checklist to count the final omissions before submission. Consistency: The number of times all formats met standards, divided by the total number of times. Cost: Subscription fees, token fees, and one-time investment in hours; amounts must come from your own bills.
The ins and outs of cost: Where tokens come from
Many people only see the total token cost without knowing where it comes from. Here are three key points. First, what you send in each time is not just your last sentence, but the entire context: the script, the dossier, all previous questions and answers, plus this new sentence. The entire stack is resent in every round, so the longer the conversation, the more is sent each time. Second, the output is measured separately, usually at a higher unit price, based on your current version. Third, here are two cost-saving methods: start a new conversation for each task; if a conversation gets too long, ask it to write a summary first, then start a new conversation with that summary. The script and dossier should be concise, but the guardrail clauses must not be deleted just to save costs. The number of tokens per Chinese character varies by provider, so do not copy English conversion rates, and do not calculate costs using ratios provided by others.
Ten identical questions, two different conversation approaches
Use an illustrative table to see the trend. Assume the script and dossier total 10 portions, with each question and answer counting as 1 portion, counting only the input. In a single conversation with ten rounds, Round 1 is 11 portions, and by Round 10 it reaches 29 portions, totaling 200 portions for ten rounds. Starting a new conversation for each of ten tasks, with 11 portions per round, totals 110 portions. The total input for a long conversation is nearly double that of starting new conversations, and it increases as the conversation progresses. These are illustrative units, not actual token counts. This assumes the ten questions are independent; if they depend on each other, keep them in the same conversation.
Lin Yue's data chain: Three sets of numbers must not be mixed
These are numbers for the same person, using the same acceptance checklist, across three stages. The first is the manual baseline: reviewing 7 past cases with the same checklist, averaging about 90 minutes, with 2.1 missed items. The second is the assistant's first trial: retesting the same 7 cases, taking about 35 minutes, with 1.3 missed items. The third is after adjusting the script: retesting 7 cases plus 4 live runs over four weeks, totaling 11 cases, taking about 28 minutes, with 0.4 missed items. Note that the 2.1 missed items refer to manual meeting minutes, not the assistant's performance. These three sets of numbers cannot be mixed.
Separate the combined numbers to see the truth
Break down the combined figures. Reviewing seven runs averages about 30 minutes, with 3 total missed items, which is somewhat optimistic because the script was adjusted before re-running. Four actual runs average about 25 minutes, with 1 total missed item (0.25 per run). Actual runs are the cleanest and most reliable, but limited to four. Combining 11 runs averages about 28 minutes, with about 0.4 missed items. The learning curve should also be documented: early runs are typically slower. Reports should include both the overall average and the stable values from later runs, not just the latter.
Industry Edition: Qilan Intelligence's Work Order Classification
Technology Industry Edition. Qilan Intelligence's Luo Xiaotang performs these tasks daily, providing a sufficient sample size without needing backtesting. The figures here are illustrative. Daily time spent dropped from approx. 90 minutes to approx. 45 minutes, including review. First response time dropped from approx. 3 hours to approx. 1.5 hours, derived from system timestamps. Misrouting rate: baseline approx. 10%; trial sampling of 20 tickets resulted in 1 error. The sample size is too small; do not state it stably dropped to 5%. Clearly state the conclusion is tentative, and plan to sample 40 tickets in the next round.
Exercise A: Organize Trial Run Data
Exercise A: Organize trial run data and verify using a calculator. Organize your trial run records into a table, separating actual runs from backtests, and calculate average time, average missed items, and consistency rate. If you lack your own data, use Lin Yue's 11-row table. Use this prompt to have it act as a data organizer. It should first repeat your data to confirm accuracy, then calculate figures for backtests, actual runs, and combined runs separately, and show the calculation process for your verification. It should directly point out missing data, not fabricate it. You must calculate the averages yourself.
Colleague Spot-Check: Don't Just Trust Your Own Missed Item Counts
You are most likely to miss items when checking your own work. Have a colleague randomly select three outputs, without looking at your numbers, and independently count them using the same checklist. Then compare the results. When Lin Yue conducted a spot check, she counted one item, while her colleague counted two. The reason was that the colleague considered a deadline listed only as 'Friday' without a specific date as a missing item, whereas Lin Yue had considered it compliant. They then updated the first item on the checklist to require a specific date and recounted according to the new standard. The rule is simple: if the total difference across three samples exceeds one item, identify the cause and standardize the criteria before drawing conclusions. Self-check against the inspection standards below.
Input-output table: Calculate cell by cell
To create an input-output table, calculate cell by cell. The baseline minus the trial average gives Lin Yue's figure: 90 minus 28, saving 62 minutes per instance. With one instance per week, the weekly savings are approximately 1.03 hours. Based on four weeks, the monthly savings are approximately 4.1 hours, which is a conservative estimate. In the monetary version, the hourly value of time is set at 100 RMB. The monthly value of saved time is 410 RMB. Subtracting the monthly subscription fee of 150 RMB, the monthly net benefit is 260 RMB. These amounts are illustrative and not real prices. Label each cell as either 'actual measurement' or 'estimate'.
Payback period: Monetary version vs. Hourly version
Payback period has two versions. Monetary version: A one-time investment of 6 hours, valued at 100 RMB per hour, totaling 600 RMB, divided by the weekly net benefit of 65 RMB, resulting in approximately 9.2 weeks. The weekly net benefit is the monthly net benefit of 260 RMB divided by 4. Hourly version: The one-time investment in hours divided by the weekly saved hours, 6 divided by 1.03, resulting in approximately 5.8 weeks. This is faster than the monetary version because it does not include subscription costs. Do not mix the two versions; clearly state which version is used in your report.
Exercise B: Have it check only the formulas
Exercise B: Do the Math. Let it do only one thing: check the formula. You fill in the numbers; it does not change them or make up prices for you. It checks whether the formula is correct, whether the units match, and whether any costs are double-counted. There are three common errors at this stage: asking the AI to look up prices, which may be outdated or incorrect; calculating only subscriptions and missing the one-time setup time; or deducting the review time twice.
Input-Output Table: Verification Criteria
Verify the Input-Output Table against these criteria. Every cell must contain a number or clearly state 'included in subscription,' with no blanks. Every number must be labeled as measured or estimated. In the cost section, ensure review time is already included in the hours used, so do not deduct it again. The payback period must be in weeks, not months, and specify whether it is the monetary or hourly version. All monetary figures must come from your own billing records. Additionally, there are three common ways to define the hourly value of time: converting from monthly salary, using internal company labor cost, or simply stating the hours without converting to money. Choose one and specify clearly.
One-Page Results Report: Six Sections
Exercise C: Write a One-Page Results Report. Structure: six sections, one or two sentences each. Section 1, One Problem: What the task was like before, and where it got stuck. Section 2, One Method: What process was used, and how humans and the Dragon Warrior divided the work. Section 3, One Set of Data: Before-and-after comparison of four metrics, specifying sample size, with live runs and backtests separated. Section 4, What Colleagues Said: Two to three direct quotes. Section 5, Unexpected Findings: One thing that differed from your expectations. Section 6, Conclusion and Next Steps: Is it worth it? (One sentence); What resources, scope, and timeline are requested. If no savings were achieved, state this truthfully and explain why.
How to Get 'What Colleagues Said': The Five Questions for Colleagues
Where does 'What Colleagues Said' in the report come from? Ask a colleague to try it, while you watch from the side without intervening or teaching. After they finish, ask them five questions: What did you use it for? At what moment did you feel it understood your work? At what moment did you feel it was talking nonsense? Will you use it in the future? Describe it in one word. Record direct quotes; do not summarize for them or beautify the language. If they say 'It was okay,' you write 'It was okay.' Direct quotes are more valuable than your summary. The AI can help you polish the report, but it cannot fabricate a colleague's genuine reaction. If you do not yet have colleague quotes before class, you can practice the structure with sample data and replace it with your own after class.
Exercise C: Have it write the report in six sections
Use this prompt to have it write a report in six sections. Its requirements are: use only the numbers you provide, keeping 'actual' or 'estimated' next to each key number; quote colleague comments exactly without rewriting, merging, or embellishing; state 'conclusions are tentative' when sample sizes are small; keep the text under 500 words. Note that it sometimes adds flattering phrases like 'significant efficiency improvement' on its own. Remove all numbers and descriptions that were not provided.
Checklist for a One-Page Report
Checklist for a One-Page Report, reviewed item by item. One page, six complete sections, every number traceable to the log, with 'actual' or 'estimated' labels. Write actual runs and backtests separately. Colleague comments must be exact quotes, not your summaries. Unexpected findings must not be written as 'consistent with expectations'; if there truly are none, it means observation was insufficient, so review the records again. The conclusion should be one sentence; next steps must include scope and timeline. If no one is willing to try it, honestly state why.
Judgment: Compare against the pre-written benchmarks.
Judgment: Compare against the pre-written success and stop-loss benchmarks. These benchmarks must be defined before trial runs and cannot be changed afterward. Lin Yue's success benchmarks are threefold: average time not exceeding half the baseline (45 minutes); average missed items not exceeding 1.0 and lower than the baseline of 2.1; no red-line incidents. The stop-loss benchmarks are also threefold; triggering any requires writing four sentences honestly. Lin Yue's results: average 28 minutes, 0.4 missed items, 0 red-line incidents. None of the stop-loss benchmarks were triggered. Judgment: Success achieved. Proceed and request small-scale rollout. The judgment has three tiers: Achieved, Partially Achieved, Not Achieved.
Quiz Question 1
Question 1: Live runs only produced four instances, which is insufficient. What should you do? A. Draw conclusions directly. B. Use historical backtesting to supplement the sample to over ten items, and present them separately in the report. C. Copy the four instances to ten. The answer is B. Backtesting only supplements samples; it cannot replace live runs. The two parts must be documented separately.
Quiz Question 2
Question 2: What does Lin Yue mean by 'Item 2.1'? A. The missing item from the assistant's first trial. B. The missing item from manual meeting minutes, i.e., the manual baseline. C. The missing item after adjusting the script. The answer is B. The baseline, the first trial, and the post-adjustment figures are three distinct numbers and must not be confused.
Quiz Question 3
Question 3: A colleague's spot check reveals a discrepancy of three items between the total of three instances and your count. What should you do? A. Trust your own count. B. First identify the cause and standardize the criteria, then draw conclusions. C. Trust the colleague's count and stop explaining. The answer is B. A discrepancy of more than one item indicates that the acceptance checklist criteria are not yet consistent.
Assignment and Project Card Updates
Assignment: The following six items are mandatory. Organized trial records: Live runs and backtesting are separated. The total must be at least 10 items; if fewer, record the actual number. Baseline records evaluated using the same acceptance checklist. Colleague spot-check records. Input-output table, with each cell marked as actual measurement or estimate. One-page results report draft, including judgment. Original quotes from at least one colleague's five-question interview. Project Card: Column 10 is closed, Column 11 contains the results, and Column 12 contains the results report draft. The AI can help polish the text, but it cannot fabricate your time logs or your colleagues' original quotes.
Next Lesson: Bringing it Back to the Company
The core of this lesson is to clarify the numbers. Remember three things: First, comparisons must be fair—same tasks, same standards, same people. Second, separate live runs from backtests; if the sample size is insufficient, state the conclusion as tentative. Third, if no savings were achieved, state that fact honestly. In the next lesson, take it back to the company: use a three-minute report to present it to your boss, and then have colleagues try it out for real. Please bring today's results report draft and judgment. Thank you for watching. See you in the next lesson. This video's narration is AI-generated.