
As AI models outgrow a single rack, memory becomes the bottleneck. When memory stays local to a server or rack, capacity sits stranded while GPUs stall waiting for data. In this video, Preet Virk, SVP & GM Photonic Fabric at Marvell, and Ravi Mahatme, Senior Director of Photonic Fabric at Marvell, explain how Marvell approaches the AI memory wall with a disaggregated, optically connected shared memory tier that spans multiple racks.
The discussion covers why KV cache placement matters as inference shifts from single-turn queries to multi-turn chats and agentic AI. It also examines how a shared memory tier moves the focus from per-GPU efficiency to cluster-level efficiency. Virk and Mahatme detail Marvell’s collaboration with Penguin Solutions, announced at Supercomputing in Chicago. They break down the Photonic Fabric Memory Module and compare two ways of connecting to the resulting appliance. They also explain what the approach means for token economics and for AI applications that are currently impractical or too costly to run.
– Why multi-rack AI models shift memory stranding from the server to the rack
– How shared KV cache keeps GPUs generating tokens instead of sitting idle during multi-turn and agentic workloads
– What Marvell builds and what Penguin Solutions delivers in the Photonic Fabric memory appliance partnership
– The capacity and bandwidth specifications of the Photonic Fabric Memory Module
– How NIC-based and Photonic Fabric chiplet connectivity compare on bandwidth, latency and placement in the memory hierarchy
– How a shared memory tier affects token throughput, token quality and power efficiency
– Why expanded context could open new AI use cases
CHAPTERS:
0:00:00 – Introduction: Memory Stranding in Multi-Rack AI Models
0:00:50 – KV Cache Sharing for Multi-Turn and Agentic AI
0:02:20 – Penguin Solutions Partnership Announcement at Supercomputing
0:02:52 – Photonic Fabric Memory Module Architecture and Specifications
0:03:25 – Division of Roles in the Penguin Appliance Partnership
0:03:53 – Connectivity Options: NIC vs. Photonic Fabric Chiplet
0:04:52 – Token Economics and Performance Gains
0:05:15 – Expanded Context Unlocking New AI Use Cases
Have a tech update that you want to brief us on? Contact info@nextgeninfra.io!
https://ngi.fyi/tech-marvell-preetravi-2
Check out other Tech Updates on our YouTube Channel (subscribe today): https://www.youtube.com/@NextGenInfra and check out our latest reports at: https://nextgeninfra.io/
