From Dev.to
vLLM Speculative Decoding: 1.8x Faster Alone, Slower at 32 Users
I turned on vLLM speculative decoding on a Friday afternoon, ran my usual one-request benchmark, and watched tokens per second jump from about 48 to about 88. I wrote "1.8x free speedup" in the team c...