AI Training Infrastructure Design Considerations

Compute, Storage, Networking, and Checkpoint Management Infrastructure Architecture Design Considerations

Technical Architecture Paper

Author: Christopher Soans
Date: October 2026

Download PDF

A vendor-neutral infrastructure perspective for scalable, resilient AI training environments

Scope: This paper presents design considerations rather than a prescriptive blueprint. Most elements use established server, networking, NVMe, storage, and redundancy practices. The principal architectural addition is a coordinated storage and checkpoint management layer with redundant centralized management processes and per-server local agents.

AI Training Infrastructure Architecture
Figure 1. AI Training Infrastructure Architecture

Executive Summary

Large-scale AI training places several very different traffic and storage demands on the same physical infrastructure. GPU collective communication is highly sensitive to latency and congestion; checkpoint movement and dataset access require sustained storage throughput; production or live traffic requires predictable service behavior; and management traffic must remain reachable during failures. A converged design can serve some environments well, but at larger scale it can also create shared congestion and failure domains.

This paper considers an architecture in which compute, storage, production/live, and management/control traffic use physically separate, redundant network fabrics. The fabrics are not assumed to have identical speeds. Each NIC, top-of-rack switch, and upstream path is sized according to the bandwidth, latency, and availability requirements of the function it serves.

Checkpoint handling is treated as a storage lifecycle rather than an immediate GPU-to-shared-storage transaction. Training servers use RAID-protected local NVMe as a staging tier. Per-server storage agents coordinate with redundant centralized storage management processes to track distributed checkpoint state, schedule transfers, validate durable checkpoint sets, manage local capacity, and coordinate restoration.

Core design principle: separate traffic by function and failure domain, then size each fabric for its own workload.

1. Purpose and Design Philosophy

The objective is not to introduce a new AI server, storage protocol, or network technology. It is to identify practical design considerations for assembling familiar infrastructure components into a training environment that limits cross-workload interference and makes checkpoint movement and recovery more predictable.

  • Use physical separation where workload criticality and scale justify the additional cost.
  • Preserve redundancy across all critical fabrics, while allowing different bandwidth specifications.
  • Keep time-sensitive GPU scheduling separate from slower cluster-wide storage scheduling.
  • Use local storage to absorb bursty checkpoint writes and decouple training progress from shared-storage latency.
  • Keep the control path separate from the high-volume checkpoint data path.
  • Treat graceful degradation and recovery behavior as first-class design considerations.

2. Reference Architecture

A training server is considered multihomed into four independent network functions. Implementations may use discrete NICs, dual-port adapters, DPUs, or other hardware arrangements; the architectural requirement is independent, redundant connectivity for each function.

Fabric

Primary Traffic

Primary Design Concern

Compute

GPU-to-GPU synchronization, RDMA, collective operations

Very high bandwidth, low latency, congestion control

Storage

Checkpoint upload/restore, datasets, model files

Sustained throughput, burst handling, storage reachability

Production / Live

Application, API, service, or live data traffic

Predictability, availability, service isolation

Management / Control

Telemetry, orchestration, controller-agent coordination

Reachability, independence, resilience

Conceptual connectivity:

TRAINING SERVER
CPU / GPU / HBM + Local NVMe RAID
        │
 ┌──────┼──────────┬──────────┐
 │      │          │          │
Compute Storage  Production  Management
NIC A/B NIC A/B   NIC A/B     NIC A/B
 │      │          │          │
ToR A/B ToR A/B   ToR A/B    ToR A/B
 │      │          │          │
Compute Storage  Production  Mgmt/Control
Fabric  Fabric    Fabric      Fabric

3. Redundancy Without Uniform Bandwidth

Redundancy and performance specification should be treated as separate design decisions. All critical training-network functions can be redundant without requiring every interface or switch to operate at the same speed. The compute fabric may justify the highest bandwidth and lowest latency, while the management fabric may require much less throughput but equally strong availability.

Design Question

Consideration

NIC sizing

Select interface capacity from measured or modeled traffic demand for that fabric.

ToR sizing

Size switching capacity and buffers for the traffic characteristics of the attached fabric.

Upstream redundancy

Avoid reconnecting redundant ToRs into a single hidden point of failure.

Failure performance

Distinguish simple connectivity survival from maintaining full performance after a component loss.

Power / cabling

Where practical, keep redundant paths independent of common power, cabling, and switch dependencies.

3A. Scale-Up GPU Interconnect and Scale-Out Compute Fabric

The compute domain should distinguish between the high-bandwidth GPU interconnect integrated into a server or rack-scale AI platform and the external network used to connect training systems beyond that local GPU domain. Vendor-integrated scale-up interconnects can provide GPU-to-GPU communication within the system or rack, while the external compute fabric provides scale-out connectivity between racks, systems, or training domains.

This distinction prevents the four-fabric model from being interpreted as four additional networks duplicating the GPU interconnect already supplied by the rack platform. The external fabrics considered in this paper are the scale-out compute, storage, production/live, and management/control networks.

Scale-up GPU interconnect  →  Scale-out compute fabric  →  Other training racks / systems

Physical separation also does not imply identical network hierarchies. Each external fabric should use the minimum topology depth and capacity necessary to satisfy its own bandwidth, latency, scale, redundancy, and failure-domain requirements. A compute fabric may require a deep, low-oversubscription leaf/spine or super-spine topology, while storage, production, or management may require fewer tiers and substantially different link speeds.

  • Compute: engineer the scale-out topology for collective communication and the required oversubscription target.
  • Storage: size leaf-to-spine capacity for dataset movement, checkpoint drain, and restore objectives, with local NVMe staging smoothing checkpoint bursts.
  • Production/live: use the topology depth and capacity required by the application traffic profile rather than copying the compute design.
  • Management/control: prioritize redundant reachability and operational independence; high compute-class bandwidth or a super-spine layer may be unnecessary.

4. Local NVMe Checkpoint Staging

Training checkpoints can represent very large distributed model and optimizer states. Rather than requiring each checkpoint to be written directly to shared storage on the critical training path, local NVMe can be used as a fast staging tier.

GPU/HBM  →  Local NVMe RAID  →  Dedicated Storage Fabric  →  Shared Durable Storage

The local tier is not intended to replace durable shared storage. RAID protects against individual local-drive failures and can preserve staged data through some hardware service events, while shared storage remains the durable recovery tier for server, rack, and broader failure scenarios.

  • Checkpoint data can be written locally at high speed before background transfer.
  • The storage network can be engineered for average drain requirements and recovery objectives rather than instantaneous checkpoint burst rate alone.
  • Local staging can buffer temporary shared-storage or storage-network interruptions.
  • A checkpoint should not be reclaimed locally until the durable copy has been validated according to policy.
  • Portable RAID metadata, encryption-key handling, and compatible replacement hardware should be considered if drive relocation is part of the recovery plan.

Optional Checkpoint Optimization

After a checkpoint has been safely staged on local NVMe, the per-server Local Storage Agent can optionally perform storage-oriented optimization before transfer to shared storage. Examples include integrity checks, packaging, lossless compression, deduplication where appropriate, and framework-approved representation changes. These functions should be policy-driven and used only when their reduction in network or storage consumption justifies their processing cost.

The training GPUs should not be required to spend valuable compute cycles performing generic storage compression. Decisions that could change numerical precision should remain under the training framework or another numerically aware component; the storage agent should not independently discard precision. This keeps the GPU focused on training while allowing the storage layer to optimize checkpoint movement when beneficial.

5. Distributed Storage and Checkpoint Management

At large scale, a single centralized process should not directly micromanage every storage operation. A distributed software control model can combine redundant centralized management processes with a lightweight local storage agent on each training server.

Centralized Management Processes

Per-Server Local Agent

Maintain global checkpoint/job state

Detect and register locally staged checkpoint components

Coordinate distributed checkpoint consistency

Monitor local RAID health and capacity

Maintain checkpoint manifests and durability state

Execute authorized upload and restore transfers

Schedule transfer priorities across the cluster

Verify local data and transfer results

Coordinate restoration and recovery

Retry/resume interrupted operations

Monitor storage/fabric health at cluster level

Reconcile state after controller or network interruption

Controller-agent coordination should use the management/control fabric, while checkpoint payloads remain on the storage fabric. This preserves a clean control-plane/data-plane separation and prevents the management service from becoming a checkpoint data bottleneck.

6. Distributed Checkpoint Consistency

A distributed training checkpoint may contain model, optimizer, scheduler, and training state sharded across many GPUs and servers. The centralized storage management layer therefore needs to understand a checkpoint as a logical set, not merely as unrelated files.

CREATING  →  LOCAL COMPLETE  →  TRANSFERRING  →  VERIFIED  →  COMMITTED / DURABLE

The central management processes coordinate, validate, and commit the distributed checkpoint. Local agents perform the data movement. A new checkpoint is not considered a valid recovery point until all required components have been received and verified and the global manifest is committed. If one component is missing or invalid, the previous committed checkpoint remains the known-good recovery point.

7. Storage-Fabric Operation

Dataset reads and checkpoint writes can share the dedicated storage fabric. Modern Ethernet and InfiniBand links are full-duplex, so opposite-direction traffic can proceed simultaneously. The architecture therefore does not require a separate checkpoint network and dataset network solely to separate upload and download directions.

Nevertheless, shared-storage controllers, metadata services, backend media, oversubscribed uplinks, or particular destinations can still become bottlenecks. The storage management system should monitor actual fabric and storage utilization and adjust transfer behavior when contention is observed rather than imposing unnecessary fixed reservations.

8. Shared-Storage Failure Behavior

Local staging provides a temporary buffer during storage interruption, but it should not be used to hide a prolonged loss of durable storage indefinitely. If shared storage remains unavailable and local staging approaches a policy-defined safety threshold, the environment should move toward a controlled training pause.

  1. Detect and confirm the storage-service failure across redundant paths.
  2. Continue staging locally while sufficient protected capacity remains.
  3. Escalate warnings as capacity or checkpoint-age thresholds are approached.
  4. Preserve known-good local and durable recovery points.
  5. Gracefully pause or quiesce affected training jobs before unsafe exhaustion.
  6. Repair and validate storage, synchronize pending checkpoints, then resume training.

A graceful pause is generally preferable to immediately powering servers off because it preserves local storage state, management reachability, telemetry, and a cleaner recovery path. Full shutdown remains an operational escalation option.

9. Checkpoint Restoration

The same controller-agent structure used for checkpoint upload can coordinate restoration. Rather than streaming restored data directly from shared storage into GPU memory, servers can first stage required checkpoint components on local NVMe, verify them, and then load them into host memory or GPU HBM as training processes become ready.

Shared Storage  →  Storage Fabric  →  Local NVMe RAID  →  Verification  →  CPU/GPU Memory

Where identical checkpoint content must be delivered to many receivers, optimized one-to-many distribution can reduce repeated transfers. IP multicast is one possible implementation, but the architecture should remain technology-neutral because many distributed checkpoints are sharded and require targeted delivery of different components to different servers or server groups.

  • Use one-to-many distribution for genuinely shared components where supported.
  • Use targeted or grouped transfer for sharded state.
  • Cache restored data locally so repeated GPU initialization does not necessarily require another network download.
  • Coordinate restore order and concurrency to avoid unnecessary storage-system hotspots.

10. Host Routing and Security Boundaries

A multihomed training server should use explicit routing and service binding so each network is used only for its intended function. A default route may reside on the primary production path, while compute, storage, and management destinations use explicit routes or appropriate policy routing.

Interface

Example Service Binding / Policy

Compute

GPU collective and RDMA services only

Storage

Checkpoint, dataset, and approved storage protocols only

Management

Controller-agent, telemetry, administration, and orchestration only

Production

Approved application and live service traffic

Routing separation should be reinforced with service-to-interface binding, host firewall policy, authentication, encryption where appropriate, and disabling unintended IP forwarding between fabrics. Physical separation then serves as both a performance boundary and a security boundary.

11. Failure-Domain Considerations

Separate redundant NICs and switches significantly limit network failure domains, but independence should be maintained beyond the server edge. Redundant ToRs that reconverge through a single upstream device, power source, or storage gateway can recreate a common point of failure.

  • Maintain independent NIC and switch paths within each critical fabric.
  • Avoid unnecessary shared upstream devices between redundant paths.
  • Use durable shared checkpoints for recovery from server or rack loss.
  • Treat complete loss of the shared storage service as an infrastructure incident rather than attempting unlimited local continuation.
  • Keep the management/control fabric available so failures in compute, storage, or production remain observable and manageable.

12. Implementation Considerations and Tradeoffs

The proposed separation is most applicable where training scale, checkpoint volume, and business criticality justify the additional hardware and operational cost. Smaller environments may reasonably use converged fabrics with QoS, segmentation, and careful capacity planning.

Consideration

Tradeoff

Four physical fabrics

Strong isolation and predictable failure domains versus more NICs, switches, optics, cabling, power, and operations.

Local NVMe RAID

Fast staging and resilience versus local capacity, write endurance, cost, and lifecycle management.

Distributed checkpoint management

Cluster-wide consistency and controlled data movement versus additional software and metadata state.

Independent fabric sizing

Cost efficiency versus the need for accurate workload characterization and growth planning.

Restore staging

Reduced dependence on shared storage during GPU initialization versus additional local-space requirements.

13. What Is Standard and What Is Added

Most components in this architecture are standard infrastructure building blocks: redundant NICs and switches, physically or logically separated networks, NVMe storage, RAID, shared storage, routing, host firewalls, and redundant control services. The architecture does not depend on a new network protocol or proprietary hardware mechanism.

The principal architectural addition is the coordinated storage and checkpoint management layer: redundant centralized management processes plus per-server local agents that collectively manage checkpoint state, transfer scheduling, consistency, durability, capacity, and restoration across the training environment. Comparable capabilities may exist in individual frameworks or storage products; this paper treats the function as an explicit infrastructure service so that the overall design remains vendor-neutral.

13A. Dedicated Training Clusters and Scheduled Capacity Utilization

Purpose-built training clusters can remain dedicated to training without requiring the same physical infrastructure to serve latency-sensitive production inference. Utilization can instead be improved through scheduled training workloads that share the cluster over time.

A cloud provider or specialized AI infrastructure operator can schedule its own foundation-model training, specialized model development, research and evaluation workloads, and customer training jobs on the same purpose-built training infrastructure. This allows organizations that do not require continuous ownership of a large GPU cluster to obtain access to high-end compute, storage, networking, power, and cooling infrastructure for defined training windows.

Dedicated training cluster  →  scheduled provider workloads / specialized models / customer training jobs  →  checkpoint and preserve state  →  next scheduled workload

This model preserves the training-oriented optimization of the scale-out compute fabric, storage fabric, checkpoint system, and maintenance schedule. It also avoids making unpredictable production-inference demand a competing resource requirement for long-running training jobs. Batch or opportunistic inference may still be an operational choice, but it is not assumed by this architecture.

  • Schedule training capacity by job, model, tenant, or reserved time window.
  • Use checkpointing to support planned workload transitions, maintenance windows, and later resumption of long-running training jobs.
  • For leased or multi-tenant capacity, isolate customer datasets, checkpoint namespaces, credentials, and network access.
  • Encrypt customer data and checkpoints where appropriate and securely reclaim or sanitize local NVMe staging before hardware is reassigned.
  • Use access controls, audit records, and tenant-aware storage policies so one customer cannot access another customer's training state.

High utilization of dedicated training infrastructure can therefore be achieved by scheduling multiple training workloads and tenants over time rather than requiring the same cluster to simultaneously serve production inference.

13B. Global QoS/Priority and Energy-Aware Training Workload Scheduling

Large training clusters are among the most power- and resource-intensive workloads in an AI environment. A global QoS/priority workload scheduler can therefore consider regional power conditions as one scheduling input while continuing to prioritize useful GPU utilization. The objective is not to idle accelerators during peak demand, but to determine which workloads should run in which datacenters and at what operating point.

New flexible workloads can preferentially be placed in datacenters with available GPU capacity and more favorable power conditions, while critical or deadline-sensitive jobs retain the resources required to meet their service objectives. Placement decisions can also consider data locality, accelerator type, network and storage capacity, thermal state, job size, expected duration, and completion deadlines.

GLOBAL QoS / PRIORITY SCHEDULER
        |
        +-- Workload Placement: priority, GPU capacity, data locality, regional power
        +-- Power/Performance Policy: workload-aware GPU operating point
        +-- Checkpoint Migration: primarily for compute/infrastructure needs

Maintain Utilization During Peak Demand

Peak regional demand does not necessarily require stopping training. Eligible lower-priority or flexible jobs can operate closer to a favorable performance-per-watt point through supported GPU power-limit or frequency controls, while higher-priority workloads can retain higher performance when required. The appropriate operating point is workload-dependent and should be based on measured performance, power, thermal, and completion-time behavior rather than a universal fixed reduction.

Placement Before Migration

Energy-aware optimization should rely primarily on initial placement and queue scheduling rather than repeatedly moving active jobs as regional conditions change. Moving a large training job may require transferring substantial checkpoint and dataset state between datacenters. Cross-datacenter checkpoint migration should therefore be driven primarily by compute or infrastructure needs—such as accelerator availability, maintenance, failures, or major capacity constraints—with power conditions acting as a secondary factor.

Separation of Scheduling Responsibilities

The global workload scheduler decides where and at what priority a training job should run. The storage/checkpoint management system remains responsible for preserving, validating, transferring, and restoring training state. Keeping these responsibilities separate avoids turning the checkpoint subsystem into a global placement engine and preserves the modularity of the architecture.

14. Summary of Design Considerations

  • Separate compute, storage, production/live, and management/control traffic when scale and criticality justify physical isolation.
  • Make every critical fabric redundant, but size NICs and switches independently for the workload they serve.
  • Use RAID-protected local NVMe as a checkpoint staging tier rather than making shared-storage latency part of every checkpoint's critical path.
  • Use redundant centralized storage management processes with local agents on each server for scalable checkpoint coordination.
  • Treat distributed checkpoint consistency as a global commit problem while keeping bulk data movement local to the storage path.
  • Use the same management framework for restoration, with local NVMe staging and optimized one-to-many distribution where appropriate.
  • Allow local staging to absorb temporary storage outages, but pause training gracefully when durable-storage loss exceeds safe policy thresholds.
  • Use explicit routing, interface binding, firewalling, and restricted forwarding to preserve the security boundaries created by the separate fabrics.
  • Maintain true end-to-end redundancy and avoid hidden common failure points upstream.
  • Distinguish vendor-integrated scale-up GPU interconnects from the external scale-out compute fabric, and avoid duplicating compute-fabric topology depth across other networks unless their workloads require it.
  • Consider scheduled multi-workload or multi-tenant use of purpose-built training clusters to improve utilization while preserving training-specific performance and failure isolation.
  • Use a global QoS/priority scheduler to keep training resources highly utilized while incorporating job priority, compute capacity, data locality, regional power conditions, and workload-specific performance-per-watt behavior into placement and operating decisions.
  • Allow the Local Storage Agent to perform optional checkpoint optimization after local staging when it produces a measurable storage or network benefit, while leaving any lossy precision decision to the training framework.

Conclusion

AI training infrastructure benefits from treating compute synchronization, storage movement, production traffic, and management as distinct operational domains. Physical separation is not required for every deployment, but at large scale it can provide predictable bandwidth, clearer failure boundaries, and simpler operational reasoning.

Checkpointing illustrates the value of this approach. Local NVMe staging can absorb bursty checkpoint creation, while a dedicated storage fabric and distributed management layer move and validate checkpoint data without placing storage scheduling responsibilities into the time-critical GPU scheduling path. The same mechanisms can coordinate restoration and preserve a clear separation between control traffic and bulk data movement.

The result is not a new hardware platform, but a set of practical design considerations for building scalable and resilient AI training environments from established infrastructure technologies, with checkpoint management treated as an explicit cluster-wide storage function.

© 2026 Christopher Soans. All rights reserved.

This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).