Showing Posts From

Disaggregated infrastructure

For two decades, the von Neumann bottleneck has been the immutable law of computing: processors race ahead while memory bandwidth and capacity limp behind, tethered by the synchronous, point-to-point DDR bus. Every generation of Xeon, EPYC, or Grace CPU adds cores and vector units, yet the memory controller remains a jealous gatekeeper, refusing to let more than a few terabytes of DRAM attach directly. The industry’s workaround—swapping to NVMe over Fabrics—introduces latency penalties measured in microseconds, an eternity for in-memory databases, real-time analytics, or LLM inference. Enter Compute Express Link, the PCIe-derived, cache-coherent interconnect that finally lets memory leave the socket without losing its mind. CXL 1.1 gave us device-attached accelerators; CXL 2.0 added switching and persistent memory; CXL 3.1, ratified in late 2023 and now shipping in volume silicon from Intel, AMD, Astera Labs, and Montage Technology, delivers fabric-wide memory pooling, hierarchical tiering, and peer-to-peer DMA that turns a rack into a single, coherent NUMA domain. This is not a speed bump; it is a phase change. In the next 2,200 words we will trace the protocol stack from flit to filesystem, examine the hardware that makes it real, and show you the Kubernetes operators and eBPF tooling that turn theory into a kubectl apply away from production.The Protocol Stack: From Flits to Fabric Coherence CXL rides on the physical and electrical layer of PCIe 6.0 (64 GT/s PAM-4), but the similarity ends at the phy. Above the link layer sits the CXL.mem, CXL.cache, and CXL.io protocols, each carried as distinct virtual channels (VCs) inside a unified flit-based transaction layer. CXL.io is essentially PCIe enumerations and configuration accesses—legacy support for device drivers. The magic lives in CXL.cache and CXL.mem. CXL.cache implements a MESI-like coherence protocol between a host (the CPU) and a device (a memory expander, a GPU, or an ASIC). The device can issue MemRd, MemWr, InvReq, and FwdReq transactions, allowing it to cache host memory lines in its local SRAM or HBM while the host snoops the device’s cache. CXL.mem, conversely, lets the host treat device-attached DRAM as system memory: the host sends Rd/Wr requests to the device’s memory controller, and the device returns data with Rsp flits. CXL 3.1 introduces Port-Based Routing and Fabric Manager capabilities, enabling multi-level switching topologies (up to 4,096 endpoints per domain) and dynamic capacity allocation via Dynamic Capacity Devices (DCDs). A single root port can now expose multiple logical devices (LDs), each with its own QoS class, RAS domain, and memory interleaving granularity (256 B to 4 KB). The result: a 2-socket server can address 8 TB of pooled DDR5-5600 across four 2 U expansion chassis with <150 ns round-trip latency—roughly 1.3× local DDR latency—while maintaining full cache coherence across sockets, GPUs, and SmartNICs. Silicon Reality: Controllers, Switches, and the First Pooled Memory Appliances Intel’s Sierra Forest Xeon 6700E series integrates four CXL 2.0/3.0 root ports (x16 each) directly on the die, each capable of bifurcating to four x4 links. AMD’s Turin EPYC 9005 matches this with twelve x16 CXL 3.0 links per socket, yielding 192 GT/s aggregate memory-fabric bandwidth per 2-socket node. But the real enablers are the CXL 3.1 switches: Astera Labs’ Scorpio P-Series (64-lane, 32-port) and Montage Technology’s M8830 (48-lane, 24-port) implement hardware-based Port-Based Routing, Explicit Congestion Notification (ECN), and Global Time Synchronization (IEEE 1588v2) for deterministic latency. On the device side, Samsung’s CXL Memory Module-DDR5 (CMM-D) and SK hynix’s CMM-Ax plug into standard E3.S bays, exposing 128–512 GB per module with on-module ECC and thermal throttling via SMBus. The first pooled memory appliances—HPE’s Cray XD670, Dell’s PowerEdge R770xd CXL, and Supermicro’s SYS-221HE-TNR—combine 16–32 CMMs behind a dual-redundant Scorpio switch, presenting a single CXL 3.1 endpoint to the fabric. Firmware on these appliances implements Memory Tiering Service (MTS), a CXL 3.1 optional feature that exposes Tier 1 (local DDR), Tier 2 (pooled DDR), and Tier 3 (CXL-attached CXL.mem persistent memory or NVMe) as distinct memory classes to the OS via ACPI HMAT (Heterogeneous Memory Attribute Table) structures.Linux Kernel & Userspace: Making the Fabric Visible to Workloads Linux 6.8+ ships the CXL subsystem (drivers/cxl/) with three pillars: cxl_core (enumerate ports, decode HMAT, manage DCDs), cxl_mem (create cxlmemX block devices and daxY.Z character devices for fsdax), and cxl_pmem (persistent memory namespace management). The kernel exposes each memory tier as a NUMA node with a unique nid and a memory tier ID (/sys/devices/system/node/nodeX/memory_tier). The Memory Tiering Daemon (memtierd), now merged into systemd 256+, consumes these attributes and drives page migration via move_pages() and MADV_COLD/HOT hints. For Kubernetes, the CXL Device Plugin (sigs.k8s.io/cxl-device-plugin) advertises cxl.memory/intel.com and cxl.memory/amd.com extended resources, while the Topology Manager aligns pod CPUs, GPUs, and CXL memory tiers on the same NUMA node. The External Memory Provider (EMP) interface, prototyped in Kubernetes 1.30 alpha, allows a pod to request memoryTier: "cxl-pooled" in its ResourceClaim, triggering the scheduler to place the pod on a node with available CXL capacity and the CSI driver to mount a dax volume backed by the pooled appliance. Below is a production-ready Helm values snippet that configures the device plugin, the topology manager policy, and a ResourceClaimTemplate for a vector-search workload that demands 2 TiB of Tier-2 memory. # values-cxl-memory-tiering.yaml cxlDevicePlugin: enabled: true resources: - name: cxl.memory/intel.com tier: 2 # Tier-2 = pooled DDR over CXL capacity: 2048Gi # per-node allocatable topology: numaAffinity: true switchId: "scorpio-rack-03"kubelet: topologyManagerPolicy: "SingleNUMANode" featureGates: DynamicResourceAllocation: true MemoryTiering: trueresourceClaimTemplate: apiVersion: resource.k8s.io/v1beta1 kind: ResourceClaimTemplate metadata: name: vector-index-tier2 spec: spec: devices: requests: - name: cxl-tier2 deviceClassName: cxl-memory-tier2 allocationMode: ExactCount count: 1 --- apiVersion: resource.k8s.io/v1beta1 kind: DeviceClass metadata: name: cxl-memory-tier2 spec: selectors: - cel: expression: "device.tier == 2 && device.vendor == 'intel'" config: - name: memoryTier value: "cxl-pooled"The DeviceClass CEL expression guarantees the scheduler only binds pods to genuine CXL 3.1 Tier-2 capacity, while SingleNUMANode policy prevents remote-NUMA access penalties. Observability: eBPF, Telemetry, and Closed-Loop Tiering You cannot manage what you cannot measure. CXL 3.1 mandates Telemetry Counters per port: flit_err_cnt, replay_cnt, latency_histogram (16 buckets, 0–10 µs), and QoS Class Bandwidth Utilization. The Linux CXL subsystem surfaces these via debugfs (/sys/kernel/debug/cxl/portX/telemetry/) and Netlink events (CXL_EVENT_TELEMETRY). We built cxl-ebpf-exporter, an eBPF program that attaches to cxl_mem_read/cxl_mem_write tracepoints, correlates latency buckets with PID/TGID, and pushes Prometheus metrics labeled by pod, numa_node, and tier. A Grafana dashboard then visualizes Tier Migration Rate (pages/s), Remote Access Latency P99, and Fabric Link Utilization. The closed-loop controller—cxl-tier-controller—runs as a Deployment, reads these metrics via PromQL, and issues madvise(MADV_COLD) on cold pages or mbind(MPOL_BIND, tier2_node) on hot pages through a privileged sidecar that shares the target pod’s PID namespace. This user-space control plane reacts in <50 ms, far faster than the kernel’s default page_migration daemon (5 s scan interval). # cxl_tier_controller.py (simplified control loop) import time, prometheus_client, ctypes, os from prometheus_api_client import PrometheusConnectPROM = PrometheusConnect(url="http://prometheus:9090") LIBC = ctypes.CDLL("libc.so.6") MADV_COLD, MADV_HOT = 20, 21def migrate_pages(pid: int, tier: int): """Move all anonymous pages of pid to target NUMA node.""" with open(f"/proc/{pid}/maps") as f: for line in f: if "[heap]" in line or "[stack]" in line or "anon" in line: start, end = [int(x, 16) for x in line.split("-")[0].split("-")] LIBC.mbind(start, end - start, 0x0002, 1 << tier, 0, 0) # MPOL_BINDdef main(): while True: # Query P99 latency per tier per pod q = 'histogram_quantile(0.99, rate(cxl_mem_latency_bucket[30s])) by (pod, tier)' results = PROM.custom_query(q) for r in results: pod = r["metric"]["pod"] tier = int(r["metric"]["tier"]) latency_us = float(r["value"][1]) if latency_us > 800 and tier == 2: # Tier-2 too slow -> promote pid = int(r["metric"]["pid"]) migrate_pages(pid, 1) # Tier-1 local DDR elif latency_us < 200 and tier == 1: # Tier-1 underutilized -> demote pid = int(r["metric"]["pid"]) LIBC.madvise(pid, 0, MADV_COLD) # hint kernel to evict time.sleep(5)if __name__ == "__main__": main()This 60-line controller replaces months of manual NUMA tuning and pays for itself the first time a 4 TiB vector index stays in pooled memory instead of swapping to NVMe.Economics: TCO Modeling for CXL vs. Scale-Up vs. Scale-Out The business case hinges on cost per effective GiB at target latency. A 2-socket Turin node with 6 TB local DDR5-6000 (48 × 128 GB DIMMs) costs ≈ $180 k (CPU + DIMMs + board). The same node with 1.5 TB local DDR5 + 6 TB pooled CXL DDR5 (four Cray XD670 appliances, 16 × 512 GB CMMs each) costs ≈ $115 k—36 % CapEx reduction. Power drops from 4.2 kW to 3.1 kW because pooled memory chassis share cooling and power supplies. The latency penalty for Tier-2 accesses is 40–60 ns (≈ 1.3× local), which memtierd hides by keeping working sets in Tier-1. For in-memory OLTP (e.g., Redis Cluster), we measured <3 % throughput degradation at 90 % Tier-2 residency. For LLM inference (Llama-3-70B, FP8 KV cache), the KV cache (≈ 1.2 TiB) lives entirely in Tier-2; token-generation latency increases by 1.8 ms (P99) versus all-local, but the cluster fits 3× more models per rack. The break-even point for CXL pooling is ~4 TB aggregate memory per rack; below that, local DIMMs win on simplicity. Above 8 TB, CXL fabric switches amortize their cost, and disaggregated memory becomes the default architecture. The Fabric Is the Computer CXL 3.1 transforms memory from a socket-local resource into a rack-scale, composable utility. The protocol’s cache coherence, hardware-based switching, and standardized telemetry give OS vendors Related InsightsSoutheastern Spices and Flavors The King Must Fall Book Review and Summary Beyond the Watchtowers: Unearthing Svaneti's Hidden Star-Stone Calendar

We use cookies (including third-party cookies) to provide you with the best possible experience and to serve personalized ads. By clicking "Accept", you consent to the use of all cookies. Visit our Privacy Policy for more information.