Infrastructure is not the goal — it is the delivery system
This section explains what infrastructure is supposed to do for AI work: make data usable, models deployable, and decisions repeatable.
Infrastructure is often misunderstood. When people hear the word, they may think of servers, GPUs, cloud accounts, or data centres. These things matter, but they are not the real goal.
The goal of infrastructure is to make useful work possible, repeatable, secure, and scalable. For AI, infrastructure is the delivery system that connects data, models, people, workflows, and decisions. It determines whether a team can move from experiment to pilot, from pilot to product, and from product to reliable operation.
For most teams, the real question is not “What is the most advanced architecture?” It is: What is the simplest architecture that supports our current AI goals safely?
It is useful to think about infrastructure through five jobs:
- Access: teams can reach the data, tools, environments, and compute they need without manual heroics.
- Scalability: the system can grow when the workload grows. For an SME, it may simply mean that the system still works when five users become fifty.
- Governance: the organization can answer who owns data, who accessed it, where it is processed, and how it is protected.
- Repeatability: experiments, deployments, and workflows can be run again without rebuilding everything from scratch.
- Optionality: enough flexibility to change model providers, hosting locations, architectures, or tools when business needs change.
Infrastructure should follow validated demand — match the investment to the stage: discovery first, scale when the value is real.
Deep Dive: The system behind the system
The questions infrastructure answers
An AI application may appear simple to the user. But behind that interaction, many components may be involved: data storage, document retrieval, model APIs, GPU servers, authentication, logging, monitoring, and deployment pipelines. Infrastructure is the foundation that lets those components work together: where data lives, who may access it, where the model runs, and how experiments are tracked. If data cannot be accessed, AI projects stall. If experiments cannot be reproduced, decisions become guesswork.
Storage, compute, and shared resources
Storage is where data, documents, model artefacts, logs, embeddings, and outputs live. Compute is where workloads run — from a laptop to a cloud VM, GPU instance, or supercomputer. Once more than one person uses compute, resource management becomes important: autoscaling, quotas, and budget limits in the cloud; job schedulers and queues on HPC. The goal is the same: make expensive shared resources usable without chaos.
Cost, governance, and build versus buy
Infrastructure creates capability, but it also creates ongoing cost — from model calls and storage to operations time. Good infrastructure includes cost visibility: teams should know what each workflow costs, not only whether it works. Governance must be implemented in infrastructure, not only in policy documents: access control, logging, audit trails, and approval workflows. And avoid infrastructure vanity projects: build or adopt what your current and near-future AI work actually needs.
Interactive task: Pick the infrastructure objective that matters most for your organization right now. You’ll get a recommended focus.
Warehouse vs lake vs lakehouse vs mesh
This section helps you understand the role of common data architectures and when each one is actually useful.
Before a model can be useful, the organization needs somewhere to store, prepare, govern, and access data. That is where data architecture comes in.
Terms such as data warehouse, data lake, data lakehouse, and data mesh are often used as if they were interchangeable. They are not. They solve different problems. Choosing between them is not mainly about which name sounds most modern. It is about what kind of data you have, who uses it, how much structure you need, and how much governance is required.
For startups and SMEs, the most important lesson is simple: do not copy enterprise architecture patterns before you have enterprise problems.
The four patterns can be compared by what problem they mainly solve:
- Warehouse — “Can we trust our structured business reporting?” Strongest when the organization needs consistent metrics and structured analytics: dashboards, finance, sales analytics, and curated model features.
- Lake — “Can we store and explore many kinds of data?” Flexible storage for raw, varied, or unstructured data: documents, logs, images, sensor data, and future use cases that are not yet fully defined.
- Lakehouse — “Can we unify analytics and AI on one governed foundation?” Combines lake-style flexibility with warehouse-style reliability and can reduce duplication — but it is not automatically necessary for every first AI project.
- Mesh — “Can the people who understand the data own and serve it?” Mainly an ownership model, not a storage technology — strongest when the bottleneck is ownership, not storage. For SMEs, the useful lesson is often simpler: make data ownership explicit.
The most practical approach is to describe the data problem first. If the problem is inconsistent reporting, start with warehouse-style modeling and metric definitions. If the problem is messy raw files and future AI use cases, start with lake-style storage plus metadata. If the problem is duplication between analytics and AI teams, consider lakehouse patterns. If the problem is that nobody owns the data, apply data mesh principles before buying another tool.
Deep Dive: Choosing a data architecture without overcomplicating it
Why this matters for AI
If the data architecture is weak, the AI system inherits that weakness. Common symptoms: teams cannot find the data they need, access rights are unclear, and models are trained on data that nobody can reconstruct. Good architecture does not guarantee good AI, but bad architecture makes reliable AI much harder. The patterns are not always mutually exclusive: a company might use a lakehouse as the technical foundation and apply data mesh principles for ownership.
Common mistakes
- Building a lake without governance: it may store everything, but help nobody.
- Using a warehouse for every kind of data: it is not always the best home for raw documents, images, or logs.
- Calling a storage platform a data mesh: buying a tool does not create a mesh if domains do not own their data.
- Choosing lakehouse because it sounds modern: it still requires governance and operating discipline.
- Ignoring small-scale simplicity: a small curated warehouse or a well-managed document repository is often enough to start.
A practical maturity path
- Stage 1 — Organize: identify where important data lives, who owns it, and which sources are authoritative.
- Stage 2 — Curate: create clean reporting tables or a trusted document collection.
- Stage 3 — Govern: add metadata, access rules, versioning, and quality checks.
- Stage 4 — Scale: adopt warehouse, lake, or lakehouse tooling as workloads grow.
- Stage 5 — Distribute ownership: apply data mesh principles when multiple domains must serve reusable data products.
Interactive task: Choose the architecture that sounds closest to your current needs. You’ll get a plain-language recommendation.
Hosting models: managed cloud, self-hosted, or hybrid
This section focuses on where models actually run — and the trade-offs between speed, cost control, and strategic independence.
For many teams, model hosting decisions are driven by convenience. But hosting is also about data movement, latency, privacy, vendor dependence, and operational burden.
A model can be technically excellent but poorly hosted. If it is too slow, too expensive, unavailable at peak times, difficult to update, or not allowed to process the required data, the application will fail in practice.
The fastest hosting path is usually a managed model API or managed cloud AI service. The provider operates the model infrastructure; your application sends requests and receives responses. The downside is that convenience creates dependency: on the provider’s availability, pricing, rate limits, and terms of service.
Self-hosting means that your organization runs the model itself. It gives more control over where data is processed, which model version is used, how endpoints are secured, and how costs are managed. But self-hosting also means your team owns the operational burden: provisioning, deployment, monitoring, failures, patching, and safe updates.
Hybrid hosting means using more than one hosting approach — often the most realistic path once an organization moves beyond early experimentation. The danger is complexity: hybrid systems require clear routing rules, consistent security, and people who understand the whole picture.
- Choose managed cloud or model APIs when speed of development is the priority, the use case is still being validated, usage is uncertain, and the provider’s data terms are acceptable.
- Choose self-hosting when data sensitivity or sovereignty is a major concern, usage is high and predictable, and you have the operational skills to run the system.
- Choose hybrid when different workloads have different risk and performance profiles, or you want a gradual path from API-based prototypes to controlled production hosting.
For many SMEs, batch prediction is underused: if a report, forecast, or classification can be computed overnight, batch hosting may be simpler, cheaper, and more robust than real-time inference.
Deep Dive: Operating a hosted model in practice
Watch-outs for managed hosting
Managed hosting should not be treated as “free infrastructure”. It shifts operational burden to the provider, but it does not remove the organization’s responsibility. Key watch-outs: what data is sent to the provider, whether prompts and outputs are stored, cost variability when usage grows, rate limits at peak demand, and behaviour changes when the provider updates the model. A practical approach is to separate application logic from provider-specific logic, so that changing models does not require rewriting the entire system.
Latency and cost have unit economics
End-to-end latency includes more than model inference: retrieval, prompt construction, and tool calls all add time. A faster model does not help much if retrieval is slow. Practical improvements include smaller models for simple tasks, caching, shorter prompts, and streaming responses. On cost, the question is: which hosting path gives acceptable quality at acceptable cost for this workload? A demo that costs very little for ten users may become expensive for ten thousand users.
Security, monitoring, and safe updates
A hosted model endpoint needs security controls like any other production service: authentication, rate limits, secret management, and output validation. In RAG systems, the model should only receive retrieved documents that the user is allowed to see. Monitor latency, error rate, cost per request, and user feedback. Update safely with realistic evaluation cases, canary releases, and rollback options. A hosted model is not “done” after launch. It is a living service.
Interactive task: Choose the hosting path you are leaning toward. You’ll get the main watch-outs to consider before committing.
Cloud, on-prem, and hybrid reality
This section helps you think beyond slogans. Most organizations do not need a pure answer — they need a practical combination.
Cloud, on-premises, and hybrid infrastructure are often discussed as if one of them must be the “right” answer. In reality, most organizations do not need a pure ideology. They need a practical deployment stance that fits their data, risk, skills, budget, existing systems, and AI ambitions.
For AI, the question is not simply “cloud or on-prem?” The question is: which workloads should run where, and why? Good infrastructure decisions start by classifying the workload:
- Is it experimental, pilot-stage, or production-critical?
- Does it use public, internal, confidential, personal, or regulated data?
- Does it need real-time responses or can it run in batch?
- Does it need strict audit trails or data residency?
- Who will operate and monitor it?
What cloud is good at: fast access to infrastructure and managed services. Cloud is often a good fit when speed of experimentation matters, usage is uncertain or spiky, and the organization does not want to operate hardware. But cloud is not magic: it still requires governance, cost monitoring, access control, and operational ownership.
What on-premises is good at: control. It can be attractive when data must remain inside organizational boundaries, regulatory requirements restrict external processing, or workloads are predictable and high-volume. But it also creates responsibility for hardware, capacity planning, patches, backups, and incident response.
Why hybrid exists: real organizations are mixed — legacy systems, cloud services, sensitive data, and emerging AI workloads at the same time. Hybrid is therefore not a failure to choose. It can be a rational design. But it should be deliberate.
A practical SME stance is usually staged and workload-based:
- Start with cloud or managed services when the use case is low-risk, experimental, and needs fast validation.
- Keep sensitive data controlled when personal, confidential, regulated, or customer-specific information is involved.
- Use on-premises or private environments when data locality, existing systems, or latency justify the extra operational responsibility.
- Use HPC or temporary GPU capacity for compute-heavy training, fine-tuning, simulation, or batch processing workloads.
- Hybridize deliberately only when there is a clear reason and a clear owner for the resulting complexity.
Deep Dive: When hybrid makes sense — and when it hurts
The hidden cost of hybrid: coordination and data movement
Hybrid infrastructure is not only technically complex. It is organizationally complex: different environments have different access systems, cost structures, monitoring tools, and operational teams. And in hybrid systems, data movement can be the real bottleneck: latency, egress costs, duplication, version conflicts, and uncertainty about which copy is authoritative. Ask: can we move computation to the data instead of moving data to computation?
Identity, governance, and skills across environments
Different environments may use different identity systems and permission models, while AI systems often sit across boundaries. Access control should be enforced in the system architecture, not left to prompt instructions. Hybrid also needs broader operational capability — a hybrid architecture that looks good on a diagram may be unrealistic if nobody can operate it.
Common anti-patterns
- Accidental hybrid: teams adopt tools independently; data, identity, monitoring, and costs end up fragmented.
- Cloud without cost visibility: usage, tokens, storage, and egress are only noticed after the workflow scales.
- On-prem control without operations capacity: everything stays internal, but nobody can manage it reliably.
- Portability theatre: the architecture claims to be portable, but important data, prompts, and monitoring are locked into one provider.
- Moving data because compute is elsewhere: large datasets are repeatedly copied across environments.
Interactive task: Select the pressures that apply to you. Then generate a suggested deployment stance.
HPC and EuroHPC: where they fit in the AI journey
This section explains the role of high-performance computing for demanding AI workloads — especially training, large-scale testing, and experimentation.
High-performance computing, or HPC, can sound intimidating. But HPC is becoming increasingly relevant for AI because modern AI workloads often need more compute, memory, storage bandwidth, and parallel processing than a laptop or ordinary server can provide.
For startups and SMEs, the important point is not that HPC is “the biggest possible infrastructure”. The important point is that shared European HPC infrastructure can give smaller organizations access to advanced compute that would otherwise be difficult or expensive to obtain.
HPC systems are usually shared. Users do not normally “own” a machine continuously. Instead, they request resources for a job: number of nodes, CPUs, GPUs, memory, and runtime.
Good AI candidates for HPC include:
- fine-tuning large models, especially when one GPU is not enough,
- distributed training across multiple GPUs or nodes,
- large-scale batch inference and embedding generation for large document collections,
- synthetic data generation and benchmarking across many configurations,
- combining traditional HPC simulation with ML surrogates or AI-assisted analysis.
EuroHPC JU is a European initiative to develop a world-class supercomputing ecosystem. Its AI Factories initiative offers AI-optimised HPC resources, support services, training, and technical expertise.
The EuroHPC AI Factory access modes are designed for different levels of need:
- Playground access: intended for SMEs, startups, and entry-level users who need rapid, limited access to test technologies — including access within two working days and onboarding services for new users.
- Fast Lane access: intended for users already familiar with HPC and requiring up to 50,000 GPU hours.
- Large Scale access: intended for AI models and applications requiring more than 50,000 GPU hours.
Deep Dive: Working on an HPC system in practice
HPC is not the same as cloud
In a cloud environment, a user often starts an instance and keeps it running as long as needed. In an HPC environment, a user typically logs into a login node, prepares files and scripts, and submits jobs to a scheduler such as Slurm. CINECA’s Leonardo system shows the pattern: login nodes for lightweight preparation — not heavy training — compute nodes for jobs, partitions, walltime limits, and shared filesystems. HPC rewards preparation: batch scripts for repeatability, small tests before scaling up.
Making the workload HPC-ready
A workload is HPC-ready when the goal is clear, data is prepared, the code can run non-interactively, resource requirements are estimated, and small tests have been run before large jobs. HPC amplifies both good and bad preparation: a well-designed job scales, a messy job fails at scale. Adding GPUs does not automatically make training faster — scaling depends on communication, batch size, and data loading. Test scaling incrementally: one GPU, then one node, then multiple nodes, measuring at each step.
When HPC is the wrong choice
HPC may be the wrong first choice when the use case is still vague, the data is not prepared, the main problem is data quality rather than compute, or the workload requires always-on low-latency production serving. Free access for eligible SMEs does not mean resources are unlimited: test small before scaling, stop failed runs quickly, and avoid idle GPUs.
Interactive task: Choose the AI activity where HPC would be most relevant for you. You’ll get a practical interpretation.
Infrastructure builder
Use this simplified builder to create a realistic infrastructure stance and see whether it looks balanced, fragile, or overbuilt.
After discussing data architecture, hosting models, cloud versus hybrid choices, and HPC, the practical question becomes: how do you turn all of this into a concrete infrastructure plan?
For startups and SMEs, the answer should not be a long technology wish list. A useful infrastructure plan starts with the AI use case, the data involved, the required reliability, the available skills, the budget, and the next stage of maturity. The goal is to choose the smallest infrastructure foundation that supports the next safe and valuable step.
The first principle is: the workload defines the infrastructure, not the other way around. A useful first step is to describe the workload in plain language:
- Who will use the AI system, and what task or decision does it support?
- What data does it need?
- Does it need real-time responses, scheduled batch processing, or occasional experimentation?
- What happens if it is wrong, unavailable, slow, or too expensive?
A practical roadmap for startups and SMEs could look like this:
- Stage 1 — Clarify: define the use case, users, data, risk, and success criteria.
- Stage 2 — Prototype: use managed tools, small samples, and lightweight tracking.
- Stage 3 — Control: add curated data, access rules, prompt/version tracking, and basic evaluation.
- Stage 4 — Pilot: add real users, monitoring, feedback, cost tracking, and ownership.
- Stage 5 — Operationalize: add deployment discipline, security review, rollback, incident handling, and support.
- Stage 6 — Scale or specialize: add self-hosting, hybrid architecture, HPC, or platform components only where the workload justifies them.
This roadmap keeps infrastructure aligned with evidence.
Deep Dive: Building an infrastructure plan from real constraints
Choosing the data foundation
Every plan needs a data foundation — not always a large platform, but knowing where data lives, who owns it, how it is accessed, and how it is kept usable. For structured analytics it may be a warehouse or lakehouse with curated tables; for RAG, an approved document repository with metadata and access control.
Repeatability, governance, and monitoring
Repeatability is one of the biggest differences between a demo and an operational AI capability: version control for code, prompts, and configuration, documented environments, dataset snapshots, model version records, and rollback. Governance must be implemented, not merely documented: identity, access, secrets, logging, retention, and auditability — for RAG systems, enforce permissions before retrieval. Plan monitoring from the beginning: latency, cost per request, failures, guardrail triggers, and user feedback. A dashboard is only useful if someone owns the response.
Common infrastructure builder mistakes
- Choosing tools before defining the workload.
- Treating data access as an afterthought: if data cannot be found or trusted, the model choice does not matter.
- Confusing prototype success with production readiness.
- Overbuilding because large companies do it: big-tech patterns may not fit SME workloads.
- Underestimating operations: self-hosted models, hybrid architectures, and HPC workflows all require people who can operate them.
- Ignoring cost until usage grows: cost visibility should start during the pilot.
Interactive task: Build a realistic setup and generate an interpretation of the trade-offs.
Key takeaways
What matters most when choosing architecture, hosting, and compute for AI initiatives.
Before approving the next AI infrastructure step, ask your team three questions:
- What stage is this initiative in — experiment, pilot, or production — and does the infrastructure match?
- What data does it need, where will the model run, and who can access the system and its data?
- How will we monitor cost, latency, quality, and failures — and who maintains the system after the first demo?