Advanced

1. Data is no longer just an input — it becomes part of the product

In a reporting system, data is mostly consumed by humans. In an ML system, data is consumed by a model. The model does not “understand” your business context in the way a person does — it learns statistical relationships from the examples it receives. If those examples are incomplete, biased, outdated, poorly labelled, or inconsistent, the model will learn from that. This is why “we have data” is not enough. For ML, you need data that is usable for learning: sufficiently clean, representative, documented, versioned, and connected to a clearly defined objective.

2. Evaluation is not a final step — it is part of the system

In many software projects, testing happens shortly before release. In ML and AI systems, evaluation has to happen repeatedly: before development, during experimentation, before deployment, after deployment, and whenever the system changes. Good evaluation starts with a baseline — if the ML system cannot reliably beat a simple baseline in the situations that matter, it may not be worth operationalizing yet. Evaluation should also separate different questions:

• Model quality: does the model make useful predictions or outputs?
• Business quality: does it improve the decision or workflow we care about?
• Operational quality: is it reliable, fast enough, secure enough, and affordable?
• Risk quality: does it behave acceptably in edge cases, sensitive cases, and failure modes?

3. Production data is a signal — and ML systems create feedback loops

A common ML failure pattern is that a model performs well during development but disappoints in production, because the data used for training does not match the data the model sees after deployment. Production data tells you whether the model still fits the real world; if you do not collect and monitor that signal, degradation becomes invisible until users lose trust. Once deployed, a model can also influence the very data it later learns from: a recommendation model changes what users see, which changes what they click, which changes future training data. Teams should intentionally decide what feedback they want to capture, how reliable it is, and how it will be used.

1. What should be tracked?

The dataset is often the most important part of an ML experiment: track the dataset version or snapshot date, source systems, date range, filters applied, label definitions, and the train/validation/test split strategy. For code, small changes can have large effects — track the git commit, branch, and dependency versions where relevant. Configuration covers the choices that shape the experiment without changing the code: model type or checkpoint, hyperparameters, feature set, prompt version if using LLMs, retrieval settings if using RAG, and the random seed. Results should include more than one final score: metrics, loss curves, performance on important slices, examples of correct and incorrect predictions, and the decision made after the experiment.

2. Experiment tracking and versioning belong together

Imagine that a model shows a strong improvement. The team wants to use it as the new baseline. But then nobody can reproduce the result. The dataset has changed. The script was edited. The random seed was not recorded. The model checkpoint was overwritten. Now the team does not know whether the improvement was real. This is not a rare edge case — it is a normal failure mode in ML teams that do not track experiments systematically. Versioning should cover data versions, code versions, configuration versions, model versions, and evaluation versions. Without versioning, experiment tracking becomes a diary. With versioning, it becomes a reproducible record.

3. From a minimum viable stack to a model registry

For a single person exploring an idea, the minimum viable setup can be simple: git for code, datasets with clear names or snapshot dates, experiment notes in a README or spreadsheet. Once several people work together, shared repositories, naming conventions, configuration files, and a shared experiment table or tracking tool (MLflow, Weights & Biases, DVC) become useful — the tool matters less than the discipline. As AI work matures, a model registry separates “all experiments” from “candidate models” and “approved production models”: it records which model artefacts exist, which have been evaluated, which are approved, and which are deployed. The same principle applies to prompt-based, RAG, and agentic systems: if system behaviour depends on it, it should be traceable. The key is proportionality. Do not overbuild. But do not rely on memory either.

1. Data leakage: when the model gets a hidden cheat sheet

Data leakage happens when information that would not be available at prediction time accidentally enters the training process. The model learns from a hidden shortcut: during evaluation, performance looks excellent; in production, the shortcut disappears. Common patterns include future information in features, preprocessing before splitting, duplicates across splits, group leakage, and process leakage — for example, a churn dataset with a field that is only updated after a customer has already contacted support to cancel. The key question is: Would this information really be available at the moment the model must make the decision?

2. Labels are assumptions, not automatically truth

Labels are often treated as ground truth, but in business settings they are frequently imperfect. A “fraud” label may only exist for cases that were investigated. A “machine failure” label may be entered inconsistently by technicians. If labels are noisy, the model learns a noisy target; if labels reflect biased processes, the model may reproduce those biases. Label management needs standards: clear definitions of each label, guidelines for human annotators, checks for disagreement, and spot reviews of edge cases. Label quality is model quality — if the organization cannot define the target consistently, the model cannot learn it reliably.

3. Version and freeze datasets — synthetic data is useful, but not magic

If a dataset is used to train or evaluate a model, it should have a version. Data changes: new rows arrive, labels are updated, preprocessing logic evolves. If the dataset is simply called “training_data.csv” and overwritten repeatedly, the organization loses memory. A disciplined folder structure, immutable snapshots, and a short dataset README can already prevent many problems. Synthetic and augmented data can increase coverage and create examples for rare cases — but use them to fill specific coverage gaps, review samples manually, keep synthetic data identifiable in the dataset version, and avoid feeding unverified model outputs back into training blindly.

1. What is a feature?

A feature is an input signal used by a model. It can be:

• Raw: a value used almost as it appears, such as age, price, or product category.
• Transformed: a cleaned or normalized version of a raw value.
• Aggregated: a summary over time, such as “average order value in the last 90 days”.
• Derived: a signal calculated from several fields, such as “days since last purchase”.
• Learned: a representation learned by another model, such as an embedding.

In foundation-model applications the idea of “feature” is broader: a retrieved document, an embedding vector, a prompt template, or a tool result can all act like features because they shape the model’s output. The language may change, but the principle remains: control the inputs that shape model behaviour.

2. Point-in-time correctness: using only what was known then

When you create training examples from historical data, each example should only use information that would have been available at that moment in time. This is especially important for rolling averages, recent activity counts, customer status, risk scores, and profiles that are updated over time. The question to ask: At the time the model would have made this prediction, would this feature value already have been known? If the answer is no, the feature introduces leakage — it may make the model look better in testing than it will be in real life.

3. Keeping feature logic consistent — and monitoring feature quality

There are three common approaches: put preprocessing inside the model (simple, but harder to reuse), use a shared, versioned transform function for both training and production (often enough for startups and SMEs), or use a feature store — a central system for managing, computing, storing, and serving features. Features can also break even when the model is unchanged: a source system changes field names, a sensor starts sending null values, a pipeline runs late. Monitor freshness, missingness, validity, distribution changes, schema changes, and cardinality. If performance drops and a key feature has silently changed, the first response may be to fix the pipeline — not to retrain the model.

1. The model lifecycle has more stages than “train and deploy”

A more realistic lifecycle: problem definition, data preparation, experimentation, evaluation, packaging, deployment, monitoring, and updating. Each stage creates artifacts — datasets, code, configurations, model files, evaluation reports, logs, and decision records. These artifacts are not administrative clutter; they are how the organization remembers what it built and why. Without this lifecycle view, teams often end up with “model islands”: one-off notebooks, undocumented deployments, unclear ownership, and models nobody fully understands a few months later.

2. Packaging and the model registry

A deployed model is rarely just a model file. The production artifact should include, or at least reference, everything needed to run the model consistently: preprocessing code, feature definitions, prompt templates, dependency versions, configuration and thresholds, and the evaluation report that justified deployment. A model registry connects the model to its context: What is this model supposed to do? Who owns it? Which dataset and code version created it? Which evaluation results justified deployment? What is the rollback path? If the only answer is “the model file is in storage”, the organization has a maintenance problem. For LLM-based systems, the deployment record should also capture prompt and system instruction versions, retrieval configuration, embedding model, tool definitions, and guardrail settings.

3. Continuous evaluation before continuous retraining — rollback is a product requirement

Retraining is not always the right first response. If performance drops because a source system broke, a field changed, or labels became inconsistent, automatic retraining may hide the root cause. A deployed model should first produce evaluation signals: samples of inputs and predictions, actual outcomes when they become available, performance by important segment, human overrides, and cost. And every production model should have a rollback plan decided before deployment, not during a crisis: return to the previous model version, turn the AI feature into “recommendation only” mode, route uncertain cases to human review, or fall back to a rule-based baseline.

1. Monitoring versus observability

Monitoring tells you that something may be wrong: latency has increased, the number of missing values has doubled, predictions have become unusually extreme. Observability helps you understand what went wrong and why — the system is instrumented well enough that you can inspect logs, traces, metrics, inputs, outputs, and intermediate steps without having to guess. In AI systems with multiple components — data pipelines, feature logic, retrieval, prompts, model APIs, guardrails — observability becomes essential. And averages can hide problems: a model may perform well overall while failing for a specific product line, region, or customer type. When something goes wrong, the team should be able to inspect examples, not only dashboards.

2. Labels arrive late — proxy signals fill the gap

The most direct way to know whether a model is still good is to compare predictions with true outcomes. But in many business settings, the true outcome arrives late: a churn prediction may only be confirmed weeks later, a loan default months later. During the gap, teams need proxy signals: input distributions, feature freshness, prediction distributions, human overrides, complaints and escalations, guardrail triggers, latency and failure rates. Proxy signals do not prove that the model is correct, but they can reveal that something has changed and needs investigation.

3. Retraining triggers: scheduled, data-driven, or performance-driven

Scheduled retraining at fixed intervals is simple and predictable, but risks unnecessary retraining if nothing meaningful changed. Data-driven retraining is triggered when enough new data has arrived or input distributions have changed — but data may be new without being better. Performance-driven retraining is usually the most meaningful trigger, but it requires reliable ground truth and a delay-tolerant evaluation process. In practice, many organizations combine all three: scheduled review, data drift monitoring, and performance-based thresholds. Feedback loops add a caveat: the system changes the data it later learns from — if feedback is biased or incomplete, retraining on it may make the system worse.