Why does an AI accelerator wait for memory? Inside high-bandwidth memory (HBM)

High-bandwidth memory (HBM) sits beside an AI accelerator and feeds the units that do the arithmetic. Why can a processor still spend time waiting for data, even when the model fits in memory? Follow a token from the interconnect of Huawei’s Atlas 960E to the weights an expert needs to read. Then look at an experiment on an NVIDIA A100 in which the same attention calculation finished sooner despite doing more arithmetic. The difference lay in how often intermediate results crossed the boundary between the processor and its memory. Specifications from AMD and Google help distinguish how much data a chip can hold from how quickly it can read them. For someone using a language model, the final test is whether these changes shorten the wait between tokens or let more people receive timely answers.