How Cerebras Deploys Trillion-Parameter AI Fashions

The whole variety of concurrent requests represents the variety of simultaneous requests transferring by the pipeline at any cut-off date. Unlike a GPU batch, these requests don’t look ahead to each other to complete, they stream by the pipeline constantly.

Of course, neither concurrency nor batch measurement tells you precisely how effectively an inference system performs. When evaluating the throughput of an inference system, the extra significant metric is whole token throughput per unit of energy (e.g. tokens per second per megawatt), when measured on the identical output technology pace.

How this scales past 3T parameters

The identical method extends to bigger fashions: calculate the reminiscence wanted for weights and request state, distribute the mannequin throughout wafers, and join them right into a pipeline. Each extra wafer contributes compute and on-chip reminiscence, permitting us to accommodate fashions that exceed the capability of any single chip.

Thanks for studying with us! Interested in deploying a mannequin on Cerebras or constructing with our inference cloud?

Designs by Halley Chang. Thanks to Atharva Talpade, Zhenwei Gao, Angela Yeung, Jessica Liu, Joyce Er, Emma Call, and Rohan Gupta for his or her overview and suggestions.