Posts Tagged 'AI'
-
January 21, 2025
AI at Scale: A Special Report
By Kirt Zimmer, Head of Social Media Marketing, Marvell
Marvell’s business is accelerated infrastructure for the AI era, which is a fast-evolving space that can occasionally confuse even the most earnest student. To help you keep up, we’ve partnered with VentureBeat to explore a multitude of content about that subject:
- Build or buy? Scaling your enterprise GenAI pipeline in 2025
Enterprise leaders are debating whether to buy AI tools, build their own, or some combination of the two. Companies like Wayfair and Expedia offer valuable insights for organizations looking to scale LLMs effectively. - Purpose-built AI hardware: Smart strategies for scaling infrastructure
Custom AI hardware is the unsung hero of scalable AI infrastructure, helping to tackle a range of issues including performance, cost, and security. For enterprises looking to transition in this rapidly evolving landscape, there’s some great advice here. - AI factories are factories: Overcoming industrial challenges to commoditize AI
Sixty years ago, Alabama was home to a 1.6GW coal fired power plant with the world's tallest chimney. That same site today houses a Google data center. The operations are obviously very different, but some of the infrastructure challenges are somewhat familiar. Read what AI 'factory' really means. - 4 bold AI predictions for 2025
We’ve seen plenty of “predictions for the coming year” pieces in other publications that are honestly pure fluff, but this ain’t that. If your brain is activated by inference costs, reasoning models, transformer alternatives and LLM scaling laws, you’ll appreciate that even annual predictions can be smart and thought-provoking.
- Build or buy? Scaling your enterprise GenAI pipeline in 2025
-
January 15, 2025
Next Up for Custom AI Accelerators: Co-Packaged Optics
By Michael Kanellos, Head of Influencer Relations, Marvell
Computer architects have touted the performance and efficiency gains that can be achieved by replacing copper interconnects with optical technology in servers and processors for decades1.
With AI, it’s finally happening.
Marvell earlier this month announced that it will integrate co-packaged optics (CPO) technology into custom AI accelerators to improve the bandwidth, performance and efficiency of the chips powering AI training clusters and inference servers and opening the door to higher-performing scale-up servers.
The foundation of the offering is the Marvell 6.4Tbps 3D SiPho Engine announced in December 2023 and first demonstrated at OFC in March 2024. The 3D SiPho Engine effectively combines hundreds of components—drivers, transimpedance amplifiers, modulators, etc.—into a chiplet that itself becomes part of the XPU.
With CPO, XPUs will connect directly into an optical scale-up network, transmitting data further, faster, and with less energy per bit. LightCounting estimates that shipments of CPO-enabled ports in servers and other equipment will rise from a nominal number of shipments per year today to over 18 million by 20292.
Additionally, the bandwidth provided by CPO lets system architects think big. Instead of populating data centers with conventional servers containing four or eight XPUs, clouds can shift to systems sporting hundreds or even thousands of CPO-enhanced XPUs spread over multiple racks based around novel architectures—innovative meshes, torus networks—that can slash cost, latency and power. If supercomputers became clusters of standard servers in the 2000s, AI is shifting the pendulum back and turning servers into supercomputers again.
“It enables a huge diversity of parallelism schemes that were not possible with a smaller scale-up network domain,” wrote Dylan Patel of SemiAnalysis in a December article.
-
December 19, 2024
Custom, Copper and Cross-Country Connectivity: Eight Big Trends for Marvell in 2024
By Michael Kanellos, Head of Influencer Relations, Marvell
What happened in semis and accelerated infrastructure in 2024? Here is the recap:
1. Custom Controls the Future
Until relatively recently, computing performance was achieved by increasing transistor density à la Moore’s Law. In the future, it will be achieved through innovative design, and many of those innovative design ideas will come to market first—and mostly— through custom processors tailored to use cases, software environments and performance goals thanks to a convergence of unusual and unstoppable forces1 that quietly began years ago.
- At Marvell AI Day, we announced that custom processors would represent 25% of the AI accelerator market by 2028 and exceed $42 billion. We also announced we had secured four designs with three of the major hyperscalers.
- In July, we unveiled StructeraTM A and Structera X,2 CXL controllers for near memory compute and increasing memory capacity, respectively. Merchant and custom versions of the chips were released at the same time, a reverse of the traditional pattern where custom chips might follow merchant ones by a lengthy lag.
- At OCP, Meta showed off FBNIC, a custom network interface developed in collaboration with Marvell to reduce the time needed to resolve network issues.3 Make that four out of four hyperscalers. See the image below.
FB NIC on display at OFC- In December, we announced a five-year collaboration that includes supplying custom AI products and other devices for AWS.4 The collaboration also encompasses performing semiconductor design tasks in the cloud, which promises to reduce the time and cost of creating new chips.
- Finally, we ended the year with an alliance for custom high bandwidth memory with Micron, Samsung Electronics, and SK hynix.5 Custom HBM can reduce chip real estate by up to 25% and increase memory capacity by up to 33%.6 Custom HBM runs against industry orthodoxy,7 writes analyst Joseph Byrne, but it could pay dividends.
-
December 10, 2024
Custom HBM: What Is It and Why It’s the Future
By Michael Kanellos, Head of Influencer Relations, Marvell
How do you get more data to the processor faster?
That has been the central question for computing architects and chip designers since the dawn of the computer age. And it’s taken on even greater urgency with AI. The greater amount of data a processor can access, the more accurate and nuanced the answers will be from the algorithm. Adding more memory, however, can also add cost, latency, and power.
Marvell has pioneered an architecture for custom high-bandwidth memory (HBM) solutions for AI accelerators (XPUs) and will collaborate with Samsung, Micron and SK hynix to bring tailored memory solutions to market. (See comments from Micron, Samsung, SK hynix and Marvell here in the release.)
Customizing the HBM element of XPUs can, among other benefits, increase the amount of memory inside XPUs by 33%, reduce the power consumed by the memory I/O interfaces by over 70%, and free up to 25% of silicon area to add more compute logic, depending on the XPU design1.
The shift—part of the overall trend toward custom XPUs--will have a fundamental and far-reaching impact on the performance, power consumption and design of XPUs. Invented in 2013, HBM consists of vertical stacks of high-speed DRAM sitting on a chip called the HBM base die that controls the I/O interfaces and manages the system. The base die and DRAM chips are connected by metal bumps.
Vertical stacking has effectively allowed chip designers to increase the amount of memory close to the processor for better performance. A scant few years ago, cutting-edge accelerators contained 80GB of HBM2. Next year, the high-water mark will reach 288GB.
Still, the desire for more memory will continue, putting pressure on designers to economize on space, power and cost. HBM currently can account for 25% of the available real estate inside an XPU and 40% of the total cost3. HBM4, the current cutting-edge standard, features an I/O that consists of 32 64-bit channels - an immense size that is already making some aspects of chip packaging extremely complex.
All About Optimizing XPU TCO
The Marvell custom HBM compute architecture involves optimizing the base HBM die and its interfaces, currently designed around standards from JEDEC, with solutions uniquely designed to dovetail with the design, characteristics and performance objectives of the host AI compute die.
Imagine that a hyperscaler wants an AI inference XPU for edge data centers squeezed into dense business districts or urban corridors. Cost and power consumption will be at a premium while absolute compute performance will likely be less important. A custom HBM solution might involve reducing the size of the AI compute die to economize on chip size and power above other considerations.
At the other end of the spectrum, an HBM subsystem for XPUs powering a massive AI training cluster might be tuned for capacity and high bandwidth. In this situation, the emphasis could be on reducing the size of the I/O interface. Reducing I/O size creates space for more interfaces on the so-called beachfront at the side of a chip and hence, boosting total bandwidth.
-
November 11, 2024
2006: The Twelve Months That Changed the Chip Industry
By Philip Poulidis,Vice President and General Manager, IoT and Mobile Business Units, MarvellThe semiconductor market is vastly different than it was a few years ago. Cloud service providers want custom silicon and collaborating with partners on designs. Chiplets and 3D devices, long discussed in the future tense, are a growing sector of the market. Moore’s Law? It’s still alive, but manufacturers and designers are following it by different means than simply shrinking transistors.
And by sheer coincidence, many of the forces propelling these changes happened in the same year: 2006.
The Magic of Scaling Slows.
While Moore’s Law has slowed, it is still alive; semiconductor companies continue to be able to shrink the size of transistors at a somewhat predictable cadence.
The benefits, however, changed. With so-called “Dennard Scaling,” chip designers could increase clock speed, reduce power—or both—with transistor shrinks. In practical terms, it meant that PC makers, phone designers and software developers could plan on a steady stream of hardware advances.
Dennard Scaling effectively stopped in 20061. New technologies for keeping the hamster wheel spinning needed to be found, and fast.
