Using AI without training a model
This section distinguishes prompting, API use, RAG, fine-tuning, and training from scratch. For many startups and SMEs, the first useful AI system will not involve training a model at all.
Core idea: modern foundation models can already perform many tasks. The technical challenge is often not “build a model”, but “give the model the right context, instructions, constraints, and evaluation loop”.
A helpful rule is: use the lightest approach that solves the real problem reliably enough. Do not start with model training if a better prompt is enough. Do not fine-tune if the model mainly lacks access to your current documents. Do not build an autonomous agent if a controlled workflow with human review is safer and easier to maintain.
Match the architecture to the failure mode. The most useful decision question is not “Which AI technique is best?” It is: what kind of failure are we trying to fix?
- The model misunderstands the task: improve instructions, examples, and output format.
- The model lacks (or has outdated) company-specific information: use RAG or tool access to retrieve the right information.
- The model answers in the wrong style or structure: improve prompt design first; consider fine-tuning if the pattern must be repeated at scale.
- The model needs to perform actions: add tools carefully, starting with read-only or human-approved workflows.
Deep Dive: Choose the lightest effective adaptation path
1. The work shifts from the model to the system around it
Modern foundation models are already useful for summarization, drafting, classification, extraction, translation, code assistance, and question answering. The work shifts from “building the model” to “building the system around the model”: clear instructions, relevant business context, structured outputs, retrieval from company documents, access control, human review for sensitive cases, and evaluation. Training a model from scratch is rarely the right starting point — the competitive advantage is usually the workflow, proprietary knowledge base, integration quality, and user experience, not the base model itself.
2. Agents and tools: when AI can act, not just answer
An application becomes more powerful when the model can use tools — search a database, query a CRM, create a ticket. Tool use changes the risk profile: a model that only drafts text can be reviewed before use; a model that sends emails or triggers payments can affect the real world. A safe design often starts with read-only tools, then recommendation mode (a human approves), then limited write actions with approval — and only later higher autonomy.
3. Evaluate before you escalate
Before moving from prompting to RAG, from RAG to fine-tuning, or from a workflow to an agent, the team needs evidence — otherwise architecture decisions are based on demo impressions. A lightweight start: 20–50 realistic examples with acceptance criteria and known edge cases, simple scoring (correct / partially correct / incorrect / unsafe), plus cost and latency notes. The failure pattern determines the next step: missing information → RAG; wrong style at scale → fine-tuning; unclear user intent → better workflow design.
Prompts, context, and structured outputs
This section covers instructions, context windows, temperature, output formats, versioned prompts, and why prompt changes should be treated like product changes.
Prompting is not just “asking nicely”. In production systems, prompts define behaviour: role, task, constraints, style, examples, allowed sources, output format, and escalation rules.
In real applications prompting is broader than wording: it is designing the whole information environment the model works in — practitioners call this context engineering. A useful prompt usually answers several questions for the model:
- What task should be performed — summarize, classify, extract, draft, route?
- What role should the model play — assistant, analyst, reviewer, compliance checker?
- What context should it use — user input, documents, retrieved passages, examples?
- What constraints matter — tone, allowed sources, uncertainty handling, escalation rules?
- What output format is required — bullet list, table, JSON, classification label?
- How will quality be judged — correctness, grounding, tone, safety, cost, latency?
The strong version fixes the role, the structure, the length, and what to do when information is missing — so the output is predictable enough to drop into a workflow.
Deep Dive: Context engineering, not magic wording
1. The context window is not a storage strategy
Many modern models accept very long prompts, but that does not mean you should put everything in. Large prompts cost more (many APIs charge by token), respond slower, add noise that reduces answer quality, expose more sensitive information, and are harder to debug. Giving a model an entire policy handbook may work in a demo — a better approach is usually to retrieve the few sections relevant to the question. A large context window is not a substitute for good information architecture.
2. Structured outputs turn model text into system input
Many AI prototypes fail when they move from “a human reads the answer” to “software uses the answer”. Humans can interpret messy text; software usually cannot. If the output feeds a workflow, ask for a specific schema instead of “extract the important details”:
This makes the AI output easier to validate, store, route, review, and monitor.
3. Prompt changes are product changes
Changing a prompt changes behaviour, just like changing code: a small wording change can alter output style, refusal behaviour, or classification boundaries. Prompts used in repeatable processes should therefore be versioned — a simple prompt register (name, use case, owner, model, output schema, date changed, evaluation notes) is enough. The same goes for variability: creative tasks may benefit from it, but when outputs feed business systems, consistency beats creativity — the model should not invent a new JSON format because it “sounds better”.
RAG: retrieval-augmented generation
This section explains documents, chunking, embeddings, vector search, retrieval, context construction, and answer generation.
RAG in one sentence: retrieve relevant information from your own knowledge base and provide it to the model as context.
RAG gives the model an “open book” before it answers — your product documentation, support articles, contracts, policies, manuals, or FAQs. You often do not need to fine-tune or train anything. But the book must be well organized, and the system must open the right page: a reliable RAG system is a pipeline, not “upload documents and chat with them”.
Retrieval found the right chunk including the exception, and the prompt required the model to answer only from it and cite the source — so the answer is correct and verifiable.
Deep Dive: RAG is a knowledge system, not just a chatbot trick
1. The three stages, and where each can fail
Indexing prepares the knowledge: documents are collected, converted to text, split into chunks, embedded, and stored. Retrieval finds the chunks most relevant to a question. Generation produces an answer grounded in them. Common failure modes are unglamorous: unapproved or outdated sources indexed, tables lost in parsing, relevant information split apart by weak chunking, semantically similar but wrong chunks retrieved, missing metadata (no filtering by region, version, or access rights), context overload, and a stale index that no longer reflects document updates.
2. Retrieval is not the same as generation
If an answer is wrong, check retrieval first: did the system find the right evidence? A fluent but wrong answer usually means the model never received the right chunk — and the fix is document cleanup, chunking, metadata, or search quality, not a bigger model.
3. Three concerns decide production trustworthiness
Knowledge-base quality — sources go stale, duplicate, and contradict each other (Section 4). Security — retrieved text is evidence, not commands; permissions belong before retrieval (Section 5). Evaluation — test retrieval and answers separately (Section 6).
RAG is one of the most practical ways for a small team to build useful AI without training a model — ideal for internal knowledge assistants, customer support, policy/compliance search, documentation, and onboarding. Scope it tightly: one use case and user group, authoritative sources with assigned owners, outdated and duplicate content removed, basic metadata, 20–50 real test questions, retrieval evaluated separately from answers, and access control before retrieval. Trying to “chat with the whole company” on day one is too broad.
Knowledge base quality for RAG
A RAG system is only as reliable as the material it retrieves. This section connects knowledge-base management to AI quality, security, and trust.
Section 3 covered the RAG pipeline — this section is about the material that flows through it. The model can only answer from what it receives: a knowledge base that is stale, duplicated, contradictory, or full of drafts hands those problems straight to the AI, often presented with confidence.
The practical consequence: a RAG knowledge base must be managed like a data product — with ownership, lifecycle rules, quality checks, access control, and feedback loops. “Chat with documents” is only reliable if the documents are reliable.
With the right metadata, retrieval can prefer the current approved Austrian version, enforce who may see it, and show users exactly where the answer came from.
Deep Dive: Your knowledge base is now part of the AI system
1. Document quality is now AI quality
Humans forgive imperfect documents — they notice a file is old or a paragraph is only a draft. A RAG pipeline is less forgiving: whatever it indexes becomes evidence the model treats as authoritative. A missing document cannot be retrieved, an outdated one yields old answers, unclear access rights leak information, poor structure breaks chunking, and missing metadata kills filtering and traceability. Start with authoritative sources that have an owner — not with “all documents”.
2. Deduplication and contradiction handling
Most organizations store the same policy in several folders and versions. Duplicates waste retrieval; contradictions are worse — if one document says refunds within 30 days and another says 14, the model may pick one, merge both, or hedge. Controls: keep only approved versions in production, mark draft/archived/current via metadata, define source-priority rules, and have the model state uncertainty when sources conflict. RAG cannot replace knowledge governance — if the organization has not decided which rule is correct, the model cannot decide for it.
3. Permissions must be enforced before retrieval
The dangerous anti-pattern is to retrieve documents first and ask the model not to reveal the sensitive parts — once content is in the context, it is already exposed. The safer principle: apply access control before retrieval. The retrieval layer must know who the user is and which documents they may access, so unauthorized content never enters the prompt — critical for HR, legal, healthcare, finance, and customer-specific material.
Guardrails, security, and technical failure modes
This section highlights hallucinations, prompt injection, access-control mistakes, unsafe tool use, sensitive data exposure, and logging risks.
Most real risks in AI applications come from the full system around the model: prompts, users, data sources, RAG pipelines, tools, logs, permissions, and workflows. A model that only drafts text in a sandbox has one risk profile; a model that can read internal documents, call tools, and trigger business actions has a very different one.
Design security around trust boundaries — the system should not treat all text equally: your system prompt is trusted application logic; a user message is not trusted; retrieved content is not trusted; a generated answer should not automatically become an irreversible business action.
The retrieved text is labelled as data, not commands, and the email tool requires separate permission and human approval — so the injected instruction cannot act.
Deep Dive: AI application risks are system risks
Module 2’s guardrails — data freshness, confidence thresholds, cost quotas, kill switches — apply here unchanged. This section adds the attack surface that is new with foundation models:
1. Prompt injection: when text tries to become instructions
Any text the model reads — a user message, a PDF, a support ticket, a web page — can contain instructions that try to override yours (“ignore your rules and…”). The defense is architectural, not polite wording: label retrieved content as untrusted data, never let it act as instructions, and gate risky actions behind separate permissions and human approval.
2. Sensitive data: prompts and logs are part of your data flow
Sensitive data can surface in prompts, retrieved context, responses, logs, and debugging traces. Settle the questions early: can users enter confidential data, is it sent to a third-party provider, is it stored in logs, who can read those logs, and is it retained or used for training? Data minimization is the right default: give the model only what the task needs — and enforce access control before retrieval and before each tool call, inherited from existing systems rather than re-invented.
3. Autonomy last: validate outputs, test the failure modes
Do not trust generated text blindly — validate outputs before they trigger anything, and treat guardrails as risk automation, not a silver bullet. Red-team the system (try to make it fail: injections, edge cases, misuse) before rollout, and monitor guardrail triggers in production. Give the AI more autonomy only when quality, permissions, monitoring, and rollback paths are in place.
Evaluation and monitoring for AI applications
This section covers correctness, faithfulness, retrieval relevance, latency, cost, refusal rate, user feedback, and production traces.
“Someone tries a few examples and decides the answer looks good” — that may be acceptable for personal productivity, but not for an AI application that supports customers, employees, or business workflows.
In a real application the model is only one component: if the answer is wrong, the cause may be retrieval, stale documents, context construction, a broken tool call, or prompt design. The key question is not “is the model good?” but “does the whole AI system reliably support the workflow it was built for?”
Because retrieval was right but the answer was wrong, the fix is prompt and grounding — a bigger model would not help. Twenty to fifty rows like this turn "it feels good" into evidence.
Deep Dive: Evaluate the system, not just the model
1. Evaluate the layers separately
Start with the use case, not a public benchmark: who uses the system, what does a good answer look like, which mistakes are unacceptable, when should it refuse or escalate? Then test retrieval and generation separately: link each test question to its expected sources and check what was actually retrieved. If retrieval fails, fix documents, metadata, chunking, or search — not the model. If retrieval was right but the answer wrong, fix prompt and grounding.
2. Structured outputs are the easiest wins
When output feeds another system — JSON, labels, extracted fields — check that it is valid, complete, uses allowed values, and handles missing information explicitly, with retry or human review when invalid. These checks are the easiest to automate and the most valuable, because invalid outputs break downstream workflows.
3. Evaluation does not stop at deployment
Monitor answer quality, retrieval failures, source freshness, invalid outputs, guardrail triggers, user feedback, cost, and latency in production, and keep traces so you can see what happened inside the system when something goes wrong. Human review needs a shared rubric to be consistent; AI-as-judge can help scale it but should not be blindly trusted. Change the system when the evidence — not a demo impression — says so.
Key takeaways
What to remember when using foundation models without training them yourself.
- Start with the lightest effective AI solution: prompting, model APIs, or RAG — add complexity only when a real failure mode justifies it.
- Prompting is application design: task, context, constraints, examples, output format — versioned like a system component, not magic wording.
- Context quality determines output quality: the model can only use the information it receives.
- RAG connects AI to your knowledge — and inherits its quality: sources, chunking, metadata, and retrieval decide reliability; the knowledge base must be managed like a data product.
- Security is system-level: prompt injection, data exposure, and unsafe tool use live in the architecture — enforce permissions before retrieval, treat retrieved text as data, not commands.
- Do not give the AI more autonomy than you can control: read-only → recommendation → approved write actions → only then more autonomy.
- Evaluate the whole application and keep monitoring it: realistic test cases before rollout; retrieval and answers separately; quality, cost, and latency in production.
A demo shows potential; evaluation shows trust. Before any rollout, ask your team three questions:
- What does the system see — and is access enforced before retrieval?
- What can it do — and which actions require human approval?
- How do we know it works — which realistic test cases and monitoring are in place?