Build AI · Technical Core
From Use Case to Decision Evidence
Take one prioritised business use case from a testable claim to a controlled AI workflow, then evaluate whether the evidence justifies another investment. The goal is not a polished demo. It is a defensible continue, adapt or stop decision.
Keep three kinds of information separate
These labels describe the current status of a claim and therefore the next action—not how optimistic the team feels.
An observed, measured or documented fact with a traceable source and context. Record what it shows and what it does not show.
A belief that needs to be true for the use case to work but has not yet been verified. Write it as a testable statement.
A relevant unknown with a named owner and a path to an answer: data check, interview, permission review or prototype test.
What this module prepares you to do
Frame a decision-ready use case
A promising idea is not yet a build brief. Make the user, moment, input, output and decision explicit.
The build question
“Can AI help sales?” is too broad to test. A bounded experiment states who needs support, when in the process, which permitted inputs are available, what reviewable output is expected and which decision or action remains human-controlled.
Shared example: after a customer meeting, an account manager provides meeting notes and approved customer context. The workflow drafts a structured CRM entry. A person reviews it before anything is stored. The business hypothesis is that preparation and documentation time can fall without reducing factual accuracy or CRM completeness.
Who needs it, when?
Account manager, immediately after a sales meeting—not “the sales department someday.”
What enters and leaves?
Meeting notes plus approved context become a structured, editable CRM draft.
What may happen?
A reviewer approves or rejects. The prototype neither messages customers nor writes unreviewed data.
Decision-maker lens
The experiment must reduce a named uncertainty. Showing that a model can generate plausible text is not enough to justify investment.
Activity · Which use case is ready for technical testing?
A company wants to improve how it decides which public tenders to pursue. Select the proposal that is sufficiently bounded for a first experiment.
Screen feasibility and choose the lightest viable pattern
Make uncertainty visible before selecting technology or treating effort as known.
Data and input
Existence, access, permission, meaning, representativeness and freshness are separate questions.
Output boundary
Define the required schema, quality checks, prohibited outputs and conditions for abstention.
Integration
Identify source and target systems. Ask whether an export or sandbox can replace a live connection initially.
Control and failure
Specify when the workflow stops, retries, flags uncertainty, asks a person or records an exception.
Ownership and operation
Name who manages access, sources, versions, review, exceptions and the decision to accept a change.
Lightest viable pattern
Try rules, then prompt plus schema, RAG or fixed tool calls. Use bounded agency or custom ML only when a concrete limitation requires it.
| Pattern | Use it when… | What enters the feasibility test |
|---|---|---|
| Rules or form | The task can be solved with deterministic logic or a template. | Rule coverage, maintenance and exception handling. |
| Prompt + structured output | Provided content must be extracted, classified or drafted into a fixed schema. | Field quality, validation, abstention and model variability. |
| RAG | The answer depends on approved knowledge outside the immediate input. RAG retrieves relevant sources before generation. | Access control, retrieval relevance, source freshness and citations. |
| Fixed tool/API workflow | The sequence is known and deterministic services must look up, calculate or store. | Permissions, interface failures, retries and idempotency. |
| Bounded agent | The next step cannot be fixed in advance and choosing among a small set of tools requires reasoning. | Allowed tools, action limits, stop conditions and traceability. |
| Custom ML | A specialised prediction task and suitable labelled data justify model development. | Label quality, baseline performance, drift, retraining and total lifecycle cost. |
Pattern glossary · open the terms you need
The patterns below are alternatives for different problems—not maturity levels that every project must climb. Open a term to see what it means, when it helps and which new uncertainty it introduces.
Prompt + structured output
A prompt is the instruction and context given to a generative model. It can include the task, constraints, examples and criteria for a good response. Structured output requires the answer to follow a fixed schema—for example named CRM fields rather than free text.
Use it when: the required facts are already present in the supplied input and must be extracted, classified or drafted. Test missing information, conflicting instructions, schema validity and model variability.
Example: turn approved meeting notes into {account_id, next_step, due_date, confidence}, leaving a field empty when the source does not support it.
Retrieval-augmented generation (RAG)
RAG first retrieves relevant passages from an approved knowledge source and then supplies those passages to a generative model. It changes the context available for a response; it does not retrain the model or guarantee that the right source will be found.
Use it when: the answer depends on sizeable, changing or access-controlled knowledge outside the immediate input. Evaluate retrieval relevance, permission filters, source freshness, citations and generation separately.
Example: compare tender requirements with the current approved capability library and cite the exact passages behind a bid/no-bid recommendation.
Fixed tool or API workflow
A tool or API is a defined capability such as looking up a record, calculating a value or storing an approved draft. In a fixed workflow, the sequence is configured in advance; the model does not decide freely what happens next.
Use it when: deterministic services must surround an AI step. Test permissions, timeouts, retries, duplicates, invalid responses and whether an action is safe to repeat.
Example: retrieve an account by ID, generate a draft, validate its schema, request human approval and only then call a test-storage API.
Bounded agent
An agent uses a model to choose a next step or tool based on the current situation. Bounded means the available tools, permitted inputs, actions, spending, number of steps and stop conditions are explicitly limited.
Use it when: the route cannot be determined reliably in advance and choosing among a small set of actions genuinely requires reasoning. If a fixed chain works, it is usually easier to test and control.
Example: choose whether to request missing tender information, search an approved capability source or escalate to a bid manager—without permission to submit a bid.
Machine learning (ML) and a custom model
Machine learning (ML) is the broader practice of learning patterns from data to make predictions or decisions. A custom model may be trained or fine-tuned for a specialised task; it is not automatically required whenever AI is involved.
Use it when: there is a stable prediction target, a suitable labelled dataset, a meaningful baseline and enough expected value to justify data preparation, training, validation, monitoring and retraining.
Example: predict equipment failure from years of labelled sensor sequences when a prompt-based workflow cannot represent the signal.
Rules or no AI
A deterministic rule, lookup, form or template produces the same result for the same input and is often easier to explain, test and operate. “No AI” is a valid technical decision.
Use it when: the conditions can be expressed directly or the cost of uncertainty is unacceptable. AI can still support a non-consequential drafting step while rules enforce the boundary.
Example: reject a record with a missing account ID through code rather than asking a model whether the identifier looks plausible.
Optional deep dive · Why “we have data” is not enough
A dataset can exist and still be unusable for the experiment. Treat existence, access, permission, meaning, representativeness and freshness as separate questions. “The notes are in the CRM” answers only the first one.
Access and permission: Can the prototype retrieve the data through an approved export or interface? Are the proposed users allowed to process it for this purpose? A screenshot, personal spreadsheet or administrator login is not a sustainable access path. Record who grants access, which fields are excluded and how test data will be minimised or redacted.
Meaning and quality: A field name does not guarantee a shared definition. “Next step” might contain a dated commitment, a vague intention or copied email text. Inspect real examples, document the intended meaning and identify deterministic checks. If two reviewers interpret the same field differently, model evaluation will be unreliable too.
Representativeness: Convenient examples create demo confidence, not feasibility evidence. Include the variation the workflow will face: short and long inputs, missing identifiers, conflicting dates, multiple languages, irrelevant content and instructions that must not be followed.
Freshness and ownership: Ask how quickly data changes, when the prototype receives updates and who corrects the source. Retrieval from an outdated capability library can be technically accurate yet produce a wrong business recommendation.
For RAG: test retrieval separately from generation. Check whether the right source passages are returned, whether permission and metadata filters hold and whether the output cites the material used. If the correct passage is never retrieved, rewriting the prompt will not solve the underlying problem.
Practical output: create a one-page data register containing source, owner, access method, permitted fields, update frequency, known quality gaps, representative sample and the test that will close each gap. This turns “we have data” into decision evidence.
Activity · Create a feasibility and pattern hypothesis
Document one known fact, one belief to test and one owned question, then choose the lightest pattern that can answer the business question.
Draw the smallest controlled workflow
Separate probabilistic generation from deterministic validation, human judgement and consequential action.
Meeting-assistant boundary
Approved notes enter the workflow. A model creates a structured draft. Required fields and permitted values are checked. Invalid drafts route to correction or manual handling. A named reviewer approves or rejects. Only an approved draft can reach the test destination, and each run records its version and outcome.
Activity · Choose the safe first workflow
Define evidence before building
Decide what the prototype must prove before an impressive output can change the success criteria.
Why this matters
The evidence plan is the acceptance brief for the prototype. It states which situations must be handled, the expected behaviour in each one, what will be measured and which result would make the team continue, adapt or stop.
Core cases · usefulness
Frequent, normal work. Expected outcome: a correct, reviewable result with all required fields.
Edge cases · brittleness
Legitimate but difficult variation. A good outcome may be a visible uncertainty flag rather than a confident answer.
Negative cases · boundary
Inputs the workflow must reject or escalate. A safe refusal can be a successful result.
| Case | Example | Expected behaviour | Decision insight |
|---|---|---|---|
| Core | Complete note with account ID | Create a structured draft for review | Can the workflow perform the frequent task? |
| Edge | Two conflicting due dates | Flag the conflict; do not invent a date | Does ambiguity create manageable review? |
| Negative | Missing permission or account ID | Stop and route to manual handling | Does the boundary hold under pressure? |
Activity · Assemble the minimum evidence plan
Select at least one core, one edge and one negative case plus two measures. Then state expected behaviour and a decision criterion.
Turn the plan into a stable evaluation
Build the prototype, then compare versions on the same cases and pre-defined outcomes.
Plan first, freeze second
Section 4 defined what should be tested. Once the first runnable workflow exists, freeze the representative cases, expected outcomes, workflow configuration and source snapshot into an evaluation version. Otherwise an improved score may only reflect easier inputs or changed expectations.
Add cases when a new failure appears, but preserve the original set. A prototype provides evidence about a specific version in a specific context; it does not prove production readiness, future economics or legal acceptability.
Hold constant
Cases, expected outcomes, pass rules, workflow/prompt/schema version and relevant source snapshot.
Record changes
Document why a case, expected outcome, model configuration or data source changed.
Limit the conclusion
“Promising enough for the next test” is often stronger and more honest than “ready to deploy.”
Activity · Define an interpretable comparison
Select everything that must be fixed or explicitly versioned:
Record decision evidence, not highlights
Averages hide failure classes. Keep the run context, version and human correction effort visible.
Did it run?
Status, latency, retries, tool errors, duplicates and completion rate.
Was context right?
Permission, completeness, retrieval relevance and missing sources.
Was it usable?
Schema pass, field correctness, harmful errors, abstention and edits.
Did work improve?
Eligible use, adoption, cycle time, user effort and outcome quality.
The evidence ledger
For each representative run, link the case and expected outcome to the exact version, result, latency, cost assumption, reviewer decision and correction effort. Record failures as diligently as passes. “82% accuracy” is not decision-ready until the unit, denominator, error severity and threshold are defined.
Activity · Add three fictitious run records
Use fictional labels and aggregated values only. This browser exercise is not an approved customer-data system.
| # | Class | Result | Latency | Review | Cost | Note |
|---|---|---|---|---|---|---|
| No fictitious runs yet. | ||||||
Expose unit economics and operating effort
Model the whole workflow and distinguish capacity created from cashable savings.
Variable system cost
Model/API calls, retrieval, external tools, compute, storage and usage-based charges.
Human cost
Preparation, review, exception handling, correction, support and time lost when the workflow fails.
Fixed and step cost
Integration, data work, security, legal/ethics review, testing, licences, monitoring, change and operation.
Capacity is not automatically cash
Time released has value only when the organisation can explain how that capacity changes real cost or produces valuable output. Do not count the same hours as both avoided cost and new sales capacity.
Activity · Calculate an explicit scenario
There are no default prices. Enter assumptions; the output is a scenario, not a forecast.
Enter all seven assumptions to calculate.
Decide: continue, adapt or stop
Pre-agreed rules protect the team from falling in love with a demo.
Evidence meets the gate
Core quality and safety thresholds pass, operating effort is plausible and remaining questions can be answered proportionately.
One focused change
A known failure suggests a bounded change to data, contract, workflow or scope. Change one important variable, version it and retest.
A constraint invalidates the case
Value is too low, permissions are unavailable, harmful errors persist, review erases benefit or a simpler solution wins.
Honest handover
Send forward the evidence that supports another investment and the critical failure, stress condition or unresolved dependency that the next stage must address. A positive average without its failure classes is not a safe decision input.
Activity · Make the decision and generate the Evidence Pack
Choose the scenario decision and complete at least seven fields.
Build AI technical summary
Eight takeaways for an evidence-based Build decision
Leadership questions
Which assumption can invalidate this case fastest? Which severe failure hides behind the average? Does review effort preserve the benefit? What result justifies another investment—and what result makes the team stop?