Fabric-Attached Shared-Memory Architecture
Scalable Fabric-Attached Shared-Memory Management, Workload Scheduling, and GPU Communication for AI Training and Inference Infrastructure
Conceptual Architecture Proposal
Author: Christopher Soans
Date: October 2026
Scope: This paper presents a conceptual architecture for evaluation and further research. It is not a prescriptive hardware design or implementation recommendation.
|
| Figure 1. Fabric-Attached Shared-Memory Architecture |
Abstract
Modern AI systems increasingly depend on large GPU clusters whose performance is shaped not only by raw compute, but also by GPU memory capacity, data locality, collective communication, scale-up bandwidth, scale-out bandwidth, and scheduling efficiency. This paper proposes a hierarchical architecture that adds a rack-level, fabric-attached shared-memory tier to conventional GPU-local HBM while preserving direct GPU-to-GPU communication for operations that benefit from the lowest possible latency.
The proposed Memory Management Node (MMN) pool is attached to the GPU scale-up fabric and provides chunk-addressable shared memory through multiple memory banks, controllers, and fabric interfaces. Per-server Multi-GPU Hardware Schedulers coordinate fast local compute and memory-placement decisions, while Global Software Schedulers coordinate workload placement, QoS, admission control, data locality, and resource allocation across servers, racks, clusters, and datacenters. NCCL or comparable collective communication libraries remain responsible for collective execution and may selectively use direct GPU paths, hierarchical collectives, in-network mechanisms, or MMN-assisted paths.
The central proposition is not to replace HBM, direct GPU communication, or existing collective libraries. Instead, the architecture explores whether decoupling GPU compute placement from exclusive GPU-local memory ownership can improve memory utilization, data reuse, activation handling, producer-consumer scheduling, MoE traffic organization, and workload locality. The proposal also defines explicit boundaries: MMN memory is not persistent storage, should not be treated as transparent HBM, does not require full rack-wide cache coherency, and should not extend transparent shared-memory semantics across datacenter WANs.
1. Motivation and Problem Statement
AI training and inference clusters are becoming increasingly hierarchical. A GPU may have very high-bandwidth local HBM, direct scale-up connectivity to peer GPUs, scale-out connectivity to remote systems, and separate access to storage. Existing collective communication mechanisms are highly optimized for many synchronization patterns, yet the architecture still commonly binds a large portion of active state to the memory of the GPU or server currently performing the work.
This proposal asks a broader question: can a rack-scale shared-memory resource, directly attached to the scale-up fabric and coordinated with hardware and software schedulers, provide an additional placement option without disrupting the paths that already work well?
- Preserve HBM as the primary hot working-memory tier.
- Preserve direct GPU-to-GPU communication for latency-sensitive operations.
- Add a fabric-attached shared-memory tier for reusable, shareable, capacity-oriented, or scheduler-managed state.
- Allow compute placement and memory placement to be coordinated but not permanently bound to one another.
- Keep collective communication vendor-neutral and compatible with existing software ecosystems.
- Coordinate QoS across compute, memory, scale-up, scale-out, and storage resources.
2. Architectural Principles
|
Principle |
Architectural meaning |
|
Complement rather than replace |
MMNs complement HBM, direct GPU communication, collective libraries, and persistent storage. |
|
Hierarchical locality |
The cost model distinguishes GPU-local HBM, rack-local shared memory, remote-rack resources, datacenter storage, and remote datacenters. |
|
Adaptive use |
The scheduler uses MMNs only when the expected benefit exceeds the latency, bandwidth, and contention cost. |
|
Chunk-addressable memory |
Shared objects are divided into manageable chunks so consumers can retrieve only the portions required. |
|
Weak, explicit consistency |
Versioning, ownership, commit states, and synchronization barriers are preferred over full CPU-style cache coherency. |
|
Vendor neutrality |
Capabilities are described abstractly; NVLink, NCCL, CXL, UALink, or other technologies may be implementation examples rather than architectural requirements. |
|
Graceful degradation |
Failure of an MMN or per-server hardware scheduler should reduce capability or performance without automatically invalidating the entire rack. |
3. Proposed System Architecture
The architecture separates global orchestration, per-server hardware scheduling, rack-level shared-memory services, collective execution, and physical transport. This separation is intended to prevent any single control layer from becoming responsible for all decisions.
|
GLOBAL SOFTWARE SCHEDULER |
3.1 GPU HBM
HBM remains the lowest-latency, highest-priority working-memory tier. Kernels should continue to use local HBM for hot data. The architecture does not assume that MMN memory can transparently substitute for HBM.
3.2 Memory Management Node Pool
The MMN pool is a rack-level logical shared-memory resource attached to the GPU scale-up fabric. The architecture does not prescribe a fixed number of MMNs per rack. Capacity and bandwidth may scale by adding modules, banks, controllers, and fabric interfaces.
- Multiple independent memory banks and memory controllers.
- Multiple scale-up fabric interfaces so aggregate bandwidth can scale with the pool.
- Chunk-addressable shared memory with directory metadata.
- Optional copy/DMA, atomic, gather/scatter, or reduction-assist engines.
- Selective replication for hot or resilience-critical chunks.
- Separate scale-out connectivity for remote-rack synchronization or transfer.
- Security domains and access controls for multi-job or multi-tenant use.
3.3 Per-Server Multi-GPU Hardware Scheduler
One Multi-GPU Hardware Scheduler is associated with each GPU server. It is not a rack-wide scheduler. Its role is to make fast local decisions using detailed knowledge of its server's GPUs, HBM state, task dependencies, thermal conditions, power state, and access to MMN resources.
- Select among local GPUs based on availability and data locality.
- Coordinate HBM/MMN placement and prefetch for active tasks.
- Account for MMN bandwidth as a schedulable resource.
- Prefer direct GPU communication when latency makes it superior.
- React quickly to local GPU or memory-access failures.
- Operate within QoS and resource constraints supplied by software orchestration.
3.4 Global Software Scheduler
The Global Software Scheduler owns policy and large-scale placement rather than fine-grained memory bookkeeping. It coordinates servers, racks, clusters, and datacenters and should itself be implemented as a resilient distributed control plane rather than a single physical controller.
- Job admission, priority, quotas, and lifecycle.
- Datacenter, cluster, rack, and server placement.
- GPU and MMN capacity commitments.
- Data locality and large-scale dependency awareness.
- QoS and fairness policy.
- Cross-datacenter placement and replication policy.
- Response to rack, server, network, power, or thermal degradation.
3.5 Collective Communication Layer
NCCL or a comparable collective library remains responsible for implementing All-Reduce, All-Gather, Reduce-Scatter, Broadcast, All-to-All, point-to-point, and other collective patterns. The library should receive capability and topology information through a vendor-neutral resource interface rather than being replaced by the MMN architecture.
|
Collective operation |
4. Fabric-Attached Shared-Memory Model
4.1 Chunking
Shared tensors and intermediate state are represented as chunks rather than as monolithic objects. Chunk sizes should be workload-aware and need not resemble CPU page sizes. This permits selective transfer, parallel access across memory banks, and progressive producer-consumer execution.
|
Tensor X |
4.2 Directory and Version Semantics
The MMN controller maintains detailed memory metadata. The per-server hardware scheduler consumes metadata for active work rather than maintaining a complete rack-scale directory itself.
|
Chunk |
Version |
State |
Location |
Policy |
|
X1 |
17 |
READY |
MMN-A |
Temporary |
|
X2 |
8 |
READ-ONLY |
MMN-B / GPU cache |
Replicable |
|
X3 |
12 |
UPDATING |
GPU17 |
Single writer |
|
X4 |
4 |
READY |
MMN-A + MMN-B |
Protected |
4.3 Consistency Without Full Cache Coherency
The architecture deliberately avoids requiring full rack-wide hardware cache coherency. GPU HBM copies are treated as temporary working copies. A scheduler or runtime validates the required version at scheduling or use boundaries. Writable chunks use ownership and commit semantics; read-only chunks may be cached broadly.
|
READY -> OWNED -> UPDATING -> COMMIT -> READY |
5. Communication Behavior
5.1 Ring and Tree Collectives
Ring and tree collectives remain valuable. A ring is a logical communication pattern in which chunks are pipelined between participants; a tree reduces logical depth through hierarchical fan-out or fan-in. Either may execute over a physical leaf-spine or fat-tree network. The MMN architecture does not assume these algorithms should be replaced.
5.2 Potential MMN Communication Advantage
For data that is reused by multiple consumers, a producer may write a chunk once into shared rack memory and allow eligible consumers to retrieve or cache it. This can remove the producer from repeated distribution and can reduce cross-rack duplication when one copy enters a destination rack and is then reused locally.
|
Conventional logical distribution: |
5.3 Where MMN Should Not Be Used
- Fine-grained, latency-critical accesses that belong in HBM.
- Direct point-to-point transfers where a peer GPU is the immediate consumer.
- Tensor-parallel communication where direct scale-up paths are superior.
- Collectives for which an optimized ring, tree, hierarchical, or in-network method is faster.
- Bulk storage traffic that does not benefit from shared-memory reuse.
6. Training Workload Behavior
Training is a primary target because it combines large memory footprints, structured dependencies, repeated collectives, activation storage, model state, and opportunities for throughput-oriented scheduling. Latency still matters in training because synchronization can leave accelerators idle, but the architecture can trade some local latency for improved aggregate utilization when the scheduler can overlap prefetch and computation.
6.1 Forward and Backward Passes
- Hot parameters and active working sets remain in HBM.
- Upcoming chunks may be prefetched from MMN while a GPU performs current computation.
- Cold activations may be offloaded to MMN and prefetched before the backward pass.
- Producer-consumer chunks may become READY progressively rather than waiting for an entire tensor.
- Direct GPU handoff remains available when it is more efficient.
6.2 All-Gather and Reduce-Scatter
The architecture is most interesting when a logical All-Gather would otherwise materialize data broadly but the next operation needs only selected chunks. Shared rack memory can make the complete logical object accessible without requiring every participant to retain every chunk in HBM. When all participants immediately require the entire object, conventional collectives may remain preferable.
6.3 All-Reduce
MMNs may optionally assist with local reductions or hold completed results, but optimized hierarchical All-Reduce is already a strong baseline. The proposal therefore treats MMN-assisted reduction as an optional capability rather than a central justification for the architecture.
6.4 Mixture-of-Experts
For MoE workloads, MMNs may provide expert queues and rack-level aggregation. Tokens for remote experts can be grouped by destination rack before scale-out transfer. This does not eliminate the payload, but it may reduce flow proliferation, provide buffering, and allow QoS-aware batching. Maximum queue delay must be controlled so batching does not create unacceptable latency.
7. Inference Workload Behavior
Inference uses many of the same mechanisms but has a different latency profile. Real-time or interactive inference should generally favor HBM and direct GPU communication. Batch inference may use MMN capacity more aggressively.
|
Workload class |
HBM preference |
MMN use |
Remote locality |
|
Real-time voice/agent |
Very high |
Limited/predictable |
Avoid remote dependencies |
|
Interactive text |
High |
Selective |
Prefer local rack/DC |
|
Batch inference |
Working set |
Significant |
Remote rack may be acceptable |
|
Background processing |
Flexible |
Aggressive |
QoS/policy controlled |
KV-cache placement is a potential inference use case: inactive or lower-priority session state could reside in MMN while active working sets remain in HBM. Any such use should be governed by QoS and prefetch behavior rather than assuming remote memory is latency-equivalent to HBM.
8. QoS and Admission Control
QoS is an end-to-end workload property rather than only a packet priority. The Global Software Scheduler establishes workload intent and resource commitments; per-server schedulers, MMN controllers, collective libraries, networks, and storage systems enforce the relevant portion.
|
WORKLOAD QoS |
8.1 Workload Priority vs. Operation Urgency
A high-priority job does not make every operation equally urgent. A GPU-blocking chunk read may be urgent while a future prefetch, replication task, or checkpoint from the same job may be less urgent. Separating workload priority from operation urgency prevents background activity from consuming resources needed on the critical path.
8.2 Admission Control
Before accepting a guaranteed-QoS workload, the Global Scheduler should evaluate existing commitments across GPU compute, HBM, MMN capacity, MMN bandwidth, scale-up capacity, scale-out capacity, storage bandwidth, power, and thermal headroom. QoS should include fairness or aging so lower-priority work is not permanently starved.
9. Multi-Rack and Multi-Datacenter Scaling
9.1 Multi-Rack
Rack-local MMNs can cache or replicate chunks so that a required chunk crosses a rack boundary once and is reused by multiple local consumers. This can reduce duplicate scale-out movement but cannot eliminate fundamental bisection bandwidth requirements when unique data genuinely must cross racks.
9.2 Multi-Datacenter
Transparent shared-memory semantics should not extend across datacenter WANs. Cross-datacenter coordination belongs to the Global Software Scheduler, which decides whether to move data, replicate state, place the workload near the data, or defer execution.
|
Locality tier |
Primary role |
|
L0 - GPU HBM |
Hot active working set |
|
L1 - Rack MMN |
Shared near-GPU memory, reuse, offload, caching |
|
L2 - Remote rack / same DC |
Selective transfer and rack-level caching |
|
L3 - Persistent storage |
Datasets, checkpoints, durable state |
|
L4 - Remote datacenter |
Workload/data placement, replication, recovery |
10. Open Capability and Topology Interface
To remain vendor-neutral, the architecture should expose capabilities rather than product names. The Global Scheduler and per-server hardware scheduler can provide constraints and resource information to the collective/runtime layer.
|
Example capability description |
An NVIDIA implementation might map these abstractions to NVLink/NVSwitch and NCCL; another implementation could use different scale-up fabrics and collective libraries without changing the architectural model.
11. Security and Isolation
A shared rack memory pool creates a security boundary that must be explicit. Multi-job or multi-tenant deployments require protected address domains, access permissions, memory sanitization on reassignment, and secure transport or memory protection where required.
- Tenant/job-scoped memory address spaces.
- Access-control enforcement in the MMN controller and fabric endpoint.
- Protection against one GPU addressing another workload's chunks.
- Zeroization or cryptographic invalidation when memory is reassigned.
- Auditable allocation and access metadata.
- Optional encryption/integrity protection according to deployment requirements.
12. Failure and Recovery Model
12.1 GPU Failure
Committed shared state may allow the per-server hardware scheduler to reassign or recompute work on another local GPU. If local capacity is insufficient, the Global Software Scheduler may reallocate the workload.
12.2 Per-Server Multi-GPU Scheduler Failure
Because the hardware scheduler is per server, its failure domain is limited to that server rather than the rack. Other servers and their hardware schedulers continue operating. The Global Scheduler can stop assigning new work to the affected server and redistribute work. A degraded conventional scheduling mode may be supported where practical.
12.3 MMN Failure
The MMN pool should contain independent modules, controllers, fabric paths, and selective replication policies. Recomputable temporary chunks need not be mirrored, while critical shared state may be replicated. Loss of MMN capacity should degrade available shared-memory functionality rather than automatically disable direct GPU operation.
12.4 Global Scheduler Failure
The global control plane should not sit in the data path. Datacenters should continue existing allocations locally during temporary loss of global coordination, while new cross-datacenter optimization or placement may pause.
13. Bandwidth and Scaling Considerations
MMN bandwidth is a first-class resource. A 64-GPU rack can generate multi-terabyte-per-second shared-memory demand even if each GPU uses the MMN for only a fraction of its memory traffic. Therefore the architecture assumes multiple active MMN modules, multiple banks/controllers, parallel scale-up interfaces, striping, hot-chunk replication, prefetch scheduling, and explicit bandwidth budgeting.
|
Average MMN demand per GPU |
64-GPU rack aggregate |
Illustrative split across 2 equal MMNs |
|
25 GB/s |
1.6 TB/s |
0.8 TB/s each |
|
50 GB/s |
3.2 TB/s |
1.6 TB/s each |
|
100 GB/s |
6.4 TB/s |
3.2 TB/s each |
These figures are illustrative rather than implementation targets. They demonstrate why MMN memory cannot be treated as ordinary server DRAM and why the number of MMNs should not be fixed by the architecture.
14. Adversarial Review and Architectural Boundaries
The concept was deliberately tested against cases where it might add unnecessary complexity or perform worse than conventional GPU infrastructure. The following boundaries emerged.
|
Challenge |
Resulting boundary |
|
Direct GPU communication |
Retain it; do not route every exchange through MMN. |
|
HBM latency/bandwidth |
MMN is a backing/shared tier, not transparent HBM. |
|
Optimized collectives |
Ring, tree, hierarchical, and in-network methods remain valid and may be superior. |
|
MMN contention |
Treat bandwidth as schedulable and scale the pool horizontally. |
|
Unpredictable access |
Prefer HBM/direct paths when prefetch and locality are poor. |
|
Metadata scale |
Use workload-aware chunk sizes and hierarchical directories. |
|
Consistency |
Use version/ownership/commit semantics instead of strong global coherency. |
|
MoE batching |
Bound queue delay through QoS and urgency. |
|
Failure domains |
Per-server hardware scheduler; distributed MMN pool; resilient global control plane. |
|
Cross-DC access |
Use placement/replication, not WAN shared-memory loads. |
|
Framework compatibility |
Keep most intelligence below the application framework. |
|
Security |
Make isolation and sanitization architectural requirements. |
15. Potential Benefits
- Rack-scale shared GPU memory capacity without replacing HBM.
- Reduced repeated cross-rack retrieval when multiple local GPUs consume the same chunk.
- Selective chunk movement instead of mandatory full-object materialization.
- Activation offload and later prefetch for training.
- Data-locality-aware GPU scheduling.
- Decoupled producer-consumer execution and progressive chunk readiness.
- Shared model/parameter caching.
- QoS-aware memory bandwidth and prefetch scheduling.
- MoE queueing and rack-level traffic organization.
- Additional recovery options when committed state exists outside a failed GPU.
- A vendor-neutral resource abstraction that can coexist with evolving fabrics and collective libraries.
16. Limitations and Research Questions
- What MMN latency and bandwidth are required for each useful workload class?
- What chunk sizes provide the best balance between selectivity and metadata overhead?
- How much activation offload can occur before MMN bandwidth becomes the bottleneck?
- Which collective operations, message sizes, and topologies benefit from MMN assistance?
- How should a collective-resource API expose locality, bandwidth, congestion, and QoS without excessive control-plane overhead?
- How should MMN capacity and bandwidth be partitioned among concurrent jobs?
- What hardware support is required for secure address translation and isolation?
- How much performance can be recovered through prefetching when MMN latency exceeds direct GPU access?
- Which existing or emerging scale-up/memory-fabric technologies can provide the required semantics most efficiently?
- How should frameworks provide optional data-lifetime or recomputability hints without becoming MMN-specific?
17. Conceptual Implementation Path
A practical research path could introduce the architecture incrementally rather than requiring a complete new GPU platform. Early prototypes could emulate MMN behavior using available memory-expansion or fabric technologies, then measure workload classes before specialized hardware is considered.
- Prototype a rack-local shared-memory service with explicit chunk APIs and no transparent coherency.
- Integrate topology and memory-locality metadata with a software implementation of the per-server scheduling policy.
- Measure All-Gather, activation offload, model caching, and MoE queueing independently.
- Add QoS and admission-control experiments under contention.
- Evaluate multi-rack caching and duplication reduction over the scale-out fabric.
- Compare direct collectives and MMN-assisted alternatives by message size and topology.
- Only after measurable benefits are demonstrated, evaluate specialized MMN controllers, hardware scheduling, and collective offload.
18. Conclusion
This paper proposes a hierarchical GPU compute and fabric-attached shared-memory architecture in which GPU-local HBM, direct scale-up communication, rack-level MMN memory, scale-out networking, and persistent storage remain distinct but coordinated resource tiers. The architecture is designed around a simple principle: GPU compute placement should not always require exclusive ownership of all useful state in the executing GPU's local memory.
The proposal does not assume that shared memory is universally faster, that collectives should be replaced, or that rack memory should extend transparently across datacenters. Instead, it gives schedulers another resource that can be used selectively when shared state, reuse, capacity relief, data locality, or producer-consumer decoupling justify it.
The concept remains intentionally implementation-neutral. Its value should ultimately be determined through simulation, prototype measurements, and workload-specific benchmarking. The architectural objective is therefore not to prescribe a particular fabric or memory technology, but to define a framework in which future GPU systems can coordinate compute, memory, communication, and QoS more flexibly while retaining compatibility with proven direct GPU and collective mechanisms.
Appendix A. Terminology
|
Term |
Meaning |
|
MMN |
Memory Management Node: a fabric-attached module or logical resource contributing shared rack memory capacity and bandwidth. |
|
Fabric-attached shared memory |
Shared memory reachable through the GPU scale-up fabric and distinct from local GPU HBM. |
|
Multi-GPU Hardware Scheduler |
Per-server hardware-assisted scheduler coordinating local GPUs, HBM locality, MMN access, dependencies, and fast local conditions. |
|
Global Software Scheduler |
Distributed orchestration layer coordinating policy, QoS, admission, placement, and resources across servers, racks, clusters, and datacenters. |
|
Chunk |
A scheduler- and memory-manageable portion of a larger tensor, model object, activation, gradient, or other state. |
|
Collective library |
NCCL or comparable software responsible for implementing GPU collective communication algorithms. |
© 2026 Christopher Soans. All rights reserved.
This work is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0).
