Is Speculative Decoding's Speedup a Hardware Problem or a Model Problem?
An engineer investigating speculative decoding speedup found that the algorithm's performance ceiling is determined by acceptance rate and cost ratio, not just hardware overhead. Testing on Apple Sili…