1. Reminiscence fragmentation
Inside fragmentation
Methods pre-allocate a big chunk of reminiscence for every request, assuming the utmost doable output size (e.g., 2048 tokens). Nonetheless, if a request solely generates a brief output, a lot of that reserved reminiscence goes unused, resulting in vital waste.
Exterior fragmentation
As a result of totally different requests reserve chunks of various sizes, the GPU reminiscence turns into scattered with unusable small gaps, making it arduous to suit new requests even when complete free reminiscence is on the market. Our sources present that in present programs, solely 20.4% – 38.2% of KV cache reminiscence is definitely used to retailer token states, with the remaining being waste.
2. No reminiscence sharing
Superior decoding methods like parallel sampling or beam search usually generate a number of outputs from a single immediate, which means they may share elements of the KV cache. Nonetheless, present programs can’t simply share this reminiscence as a result of every sequence’s KV cache is in its personal separate, contiguous block.