# NVIDIA argues full-stack codesign can ease AI’s memory challenge [Canonical reading](https://escapemeridian.com/readings/nvidia-argues-full-stack-codesign-can-ease-ai-s-memory-challenge-86899040) NVIDIA’s post argues that adding HBM alone cannot solve rising memory pressure: bandwidth limits how quickly data can reach GPUs, and extra capacity can add cost and power without being fully used. Its proposed answer is full-stack codesign, coordinating GPUs, CPUs, networking, storage and software to place data across a memory and storage hierarchy. The post describes shrinking model weights and KV cache, moving inactive context to lower-cost tiers, and keeping RAG indexes outside HBM. It says these techniques leave more HBM available for active context and concurrent users. NVIDIA also presents SemiAnalysis AgentX results as evidence that throughput per dollar has risen much faster than HBM capacity across its platform generations. Those results are tied to specified workloads and response-time conditions; the post’s account does not establish how the approach performs across other models or operating settings. For AI factory operators, NVIDIA frames the practical measure as useful tokens delivered within power and response-time limits, rather than memory capacity alone. Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims. ## Original source NVIDIA · Article Original source date: Nov 5, 2026 (2026-11-05T16:00:00+00:00). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-09T05:39:22.776Z. First observed: 2026-11-05T16:06:32.202Z. Retrieved: 2026-11-09T05:38:41.586Z. - [More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Limits The performance comparisons are NVIDIA-reported results on named SemiAnalysis AgentX workloads; the post does not establish performance across other workloads or operating settings. The post does not provide independent validation of its reported comparisons. The post describes techniques and platform results but does not quantify the separate contribution of each technique to the reported gains. ## Open question How would full-stack codesign’s reported throughput-per-dollar gains compare across workloads and operating settings beyond the SemiAnalysis AgentX results described here? ## Source-stated NVIDIA argues that increasing HBM capacity alone can be ineffective when memory bandwidth limits how quickly data reaches compute. > eventually memory bandwidth is unable to supply data as fast as compute can use it: a limit engineers call the memory wall. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated NVIDIA describes full-stack codesign as coordinating hardware and software to use the memory hierarchy more efficiently. > NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated The post says its approach reduces HBM use by shrinking model weights and KV cache, moving inactive context to other tiers, and keeping the RAG index outside HBM. > The result is less HBM used for model weights, less HBM tied up by idle context and more of it available to serve concurrent users. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated NVIDIA reports substantially higher throughput per dollar on the SemiAnalysis AgentX workloads while HBM capacity per GPU rose less. > At 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, with HBM capacity per GPU up about 3%. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated NVIDIA says AI factory operators should assess useful output against power and response-time needs, not memory capacity alone. > For AI factory operators, the metric that matters is how many useful tokens the factory can deliver within its power budget while meeting user response-time needs. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source and contributor context Amr Elmeleegy · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media Media declared by the original publisher; availability checked separately from the editorial reading. - [image · NVIDIA](https://blogs.nvidia.com/wp-content/uploads/2026/11/memory-codesign-1920x1080-1-1536x864.jpg) ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Graphics processing unit A chip that performs many calculations at once; the main hardware used to train and run AI models. - [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit) ## CPU The central processing unit: a general-purpose processor that carries out a computer's instructions. - [Definition reference](https://aws.amazon.com/what-is/cpu/) ## Model weights The learned numbers that make up a trained model; publishing them lets others run it. - [Definition reference](https://developers.google.com/machine-learning/glossary#weight) ## Retrieval-augmented generation A setup in which a model first looks up relevant documents and then answers using them. - [Definition reference](https://developers.google.com/machine-learning/glossary#retrieval-augmented-generation-rag) ## NVIDIA NVIDIA develops processors, networking and software used for AI, graphics and scientific computing. - [Explore this context](/entities/nvidia) - [Reference](https://blogs.nvidia.com/) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 0b013dc5-a415-4669-883b-3357eaaa07f2. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T05:39:22.776Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 21a9247a888059769e89b1e6704e82ac0e4473ddda5d86922dde95ad888a2a66. Source text hash: df3b54e73d827840575c274db6b14a7ead94f0b483c511a5af88b3623f773687. ## Revision and source-check history ### Version 1 — 2026-11-09T05:39:22.776Z NVIDIA’s post argues that adding HBM alone cannot solve rising memory pressure: bandwidth limits how quickly data can reach GPUs, and extra capacity can add cost and power without being fully used. Its proposed answer is full-stack codesign, coordinating GPUs, CPUs, networking, storage and software to place data across a memory and storage hierarchy. The post describes shrinking model weights and KV cache, moving inactive context to lower-cost tiers, and keeping RAG indexes outside HBM. It says these techniques leave more HBM available for active context and concurrent users. NVIDIA also presents SemiAnalysis AgentX results as evidence that throughput per dollar has risen much faster than HBM capacity across its platform generations. Those results are tied to specified workloads and response-time conditions; the post’s account does not establish how the approach performs across other models or operating settings. For AI factory operators, NVIDIA frames the practical measure as useful tokens delivered within power and response-time limits, rather than memory capacity alone. ## Original source NVIDIA · Article Original source date: Nov 5, 2026 (2026-11-05T16:00:00+00:00). Date basis: Date stated by the publisher. Added to Escape Meridian: 2026-11-09T05:39:22.776Z. First observed: 2026-11-05T16:06:32.202Z. Retrieved: 2026-11-09T05:38:41.586Z. - [More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Limits The performance comparisons are NVIDIA-reported results on named SemiAnalysis AgentX workloads; the post does not establish performance across other workloads or operating settings. The post does not provide independent validation of its reported comparisons. The post describes techniques and platform results but does not quantify the separate contribution of each technique to the reported gains. ## Open question How would full-stack codesign’s reported throughput-per-dollar gains compare across workloads and operating settings beyond the SemiAnalysis AgentX results described here? ## Source-stated NVIDIA argues that increasing HBM capacity alone can be ineffective when memory bandwidth limits how quickly data reaches compute. > eventually memory bandwidth is unable to supply data as fast as compute can use it: a limit engineers call the memory wall. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated NVIDIA describes full-stack codesign as coordinating hardware and software to use the memory hierarchy more efficiently. > NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated The post says its approach reduces HBM use by shrinking model weights and KV cache, moving inactive context to other tiers, and keeping the RAG index outside HBM. > The result is less HBM used for model weights, less HBM tied up by idle context and more of it available to serve concurrent users. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated NVIDIA reports substantially higher throughput per dollar on the SemiAnalysis AgentX workloads while HBM capacity per GPU rose less. > At 125 tokens per second per user, Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, with HBM capacity per GPU up about 3%. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source-stated NVIDIA says AI factory operators should assess useful output against power and response-time needs, not memory capacity alone. > For AI factory operators, the metric that matters is how many useful tokens the factory can deliver within its power budget while meeting user response-time needs. - [Supporting source](https://blogs.nvidia.com/blog/memory-full-stack-codesign/) ## Source and contributor context Amr Elmeleegy · Article author These are credits stated by the source; they do not establish a profile match or authorship of every passage. ## Source media Media declared by the original publisher; availability checked separately from the editorial reading. - [image · NVIDIA](https://blogs.nvidia.com/wp-content/uploads/2026/11/memory-codesign-1920x1080-1-1536x864.jpg) ## Terms and references Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement. ## Graphics processing unit A chip that performs many calculations at once; the main hardware used to train and run AI models. - [Definition reference](https://en.wikipedia.org/wiki/Graphics_processing_unit) ## CPU The central processing unit: a general-purpose processor that carries out a computer's instructions. - [Definition reference](https://aws.amazon.com/what-is/cpu/) ## Model weights The learned numbers that make up a trained model; publishing them lets others run it. - [Definition reference](https://developers.google.com/machine-learning/glossary#weight) ## Retrieval-augmented generation A setup in which a model first looks up relevant documents and then answers using them. - [Definition reference](https://developers.google.com/machine-learning/glossary#retrieval-augmented-generation-rag) ## NVIDIA NVIDIA develops processors, networking and software used for AI, graphics and scientific computing. - [Explore this context](/entities/nvidia) - [Reference](https://blogs.nvidia.com/) ## TIME TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services. - [Explore this context](/entities/time) - [Reference](https://time.com/) ## Editorial provenance Version 1. Publication ID: 0b013dc5-a415-4669-883b-3357eaaa07f2. Prepared by gpt-6-luna; grounding review by gpt-6-luna. Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-09T05:39:22.776Z. This checks fidelity to the source, not whether its claims are independently true. Editorial selection: accept; reasons: material\_development. Observation: 21a9247a888059769e89b1e6704e82ac0e4473ddda5d86922dde95ad888a2a66. Source text hash: df3b54e73d827840575c274db6b14a7ead94f0b483c511a5af88b3623f773687. ### uncertain — 2026-11-09T05:39:22.776Z — anthropic/claude-opus-5.5 Publication: 0b013dc5-a415-4669-883b-3357eaaa07f2 attribution — pass: The reading consistently attributes the argument, techniques and performance figures to NVIDIA's own blog post. It frames SemiAnalysis AgentX results as NVIDIA-presented rather than independently established. > NVIDIA intentionally designs GPUs, CPUs, networking, storage and software together to get more useful work out of memory at every layer of the AI factory. summary — uncertain: The reading faithfully captures NVIDIA's thesis: capacity is limited by bandwidth, codesign manages a memory hierarchy, and tokens per dollar or watt is the key metric. However, central terms are not explained in plain language: HBM, KV cache, RAG indexes and AI factory. A non-specialist cannot tell that HBM is the GPU's fastest, most expensive memory tier or that KV cache holds the context for active requests. The source supplies both definitions. Smallest repair: add short glosses for HBM and KV cache, and briefly gloss RAG, in the summary. > GPU high-bandwidth memory (HBM) is the fastest, most expensive and most contested tier — with model weights and KV cache competing for limited capacity. date — pass: originalPublished is 2026-11-05T16:00 UTC from the publisher page, matching the feed entry and first observation time. The draft makes no conflicting date claim. support — pass: Each claim matches a source-stated passage. The limitations correctly note that the gains are vendor-reported on named workloads, have no independent validation and are not broken down by technique. The question follows from the evidence without leading. The reading does not turn the benchmark into demonstrated real-world reliability. > On the SemiAnalysis AgentX DeepSeek V4 Pro workload, NVIDIA GB300 NVL72 delivers up to 10x higher throughput per dollar than NVIDIA H200 platform at 100 tokens per second per user media — pass: The selected media is the verified direct link to the original NVIDIA blog post, which is the most direct access to the source. ## Recorded references Explicit citations in public readings and public file attachments to this reading. Shared topics are not citations. Other artifact types need an explicit reference adapter. No public references recorded here yet. [Reference data (JSON)](/readings/86899040e99a84796d1b91754aa428c1080127286b85ce6ee1f0414d6dd6189f/references.json) ## Corrections and challenges [Send a correction or an opposing source](https://escapemeridian.com/corrections/) [JSON record](https://escapemeridian.com/readings/nvidia-argues-full-stack-codesign-can-ease-ai-s-memory-challenge-86899040.json)