I Trusted the Community and Halved My Decode Speed
Believing "speculative decoding and multimodal are incompatible," I turned off MTP and 42.8t/s dropped to 19.4t/s. Reading the source, the code only warns and proceeds, and meanwhile another trap had been armed
I decided to put a chat UI in front of the gemma server on the DGX Spark so friends could use it too. While at it I wanted image input, and since the model itself is multimodal it looked like only the server config had to change. But I took the line that speculative decoding and multimodal can’t run together at face value, and so it began as a trade that gave back the decode speed I’d gained the day before.
What I set out to do
I decided to attach a chat UI to share with friends. Open WebUI, placed behind a Cloudflare tunnel. While at it I wanted images too. Gemma 4 is a multimodal model, so only the server config needs to change.
If I enable images, does performance get bad again? Attach Open WebUI.
How I asked
Claude Code’s answer was “in vLLM, speculative decoding and multimodal input are incompatible.” A story I’d seen many times in community issues. So I dropped the MTP I’d adopted the day before and enabled images.
Single-stream decode fell from 39.5t/s to 19.4t/s. One disclosure-review JSON went from 1.9 seconds to 4.1 seconds. I accepted it as a trade: half the speed for images.
Where it broke
The moment I accepted it, the server died. The engine crashed instantly on the first structured-output (json_schema) request.
RuntimeError: sample_tokens() must be called after execute_model() returns NoneDrop MTP and vLLM automatically turns on async scheduling. Receive a structured-output request in that state and it dies. The cause was in the source. The code that re-invokes deferred sampling lived only inside a conditional on a speculative-decoding-only flag. Use async and json_schema together without speculation and that call never happens.
For the moment I forced --no-async-scheduling on to block it. When it said “this is an operationally fatal trap, I’ll record it in memory,” I asked back for the cause and got that explanation. Fixing A had armed B, and B could only be defused by reverting A.
How I fixed it
At 10:30 pm I went back to the original premise. I had it read the source to find what “incompatible” was actually based on.
What runs in vLLM when multimodal and speculative decoding are both enabled is _warn_if_multimodal(). As the name says, it prints a warning and proceeds. The only hard failure is _raise_if_mrope(), and that applies only to models using M-RoPE positional encoding. Gemma 4 doesn’t use M-RoPE.
I restored MTP and re-measured with images enabled. Single 42.8t/s, 339t/s at 16 concurrent, image input working, json_schema 3/3, tool calls 9/9. There was never a need to choose between speed and images. The 50% loss was the cost of trusting one line in an issue comment.
Lessons
- Verify “incompatible” against the source. A warning and an exception are different things. Sometimes an issue comment transcribes code that warns-and-proceeds as “not possible.”
- Removing one setting can silently flip another default. Dropping MTP turned on async scheduling. Don’t look only at what you removed; look at what changed as a result.
- Put structured output in the smoke test. There are combinations where plain chat is fine but the first json_schema request kills the server. Throw one right after the server comes up.
- Before accepting a loss, doubt the premise once more. The moment I decided “half is acceptable” was the moment I should have read the source.
- Check the agent’s confidence by asking for its basis. Both the incompatibility and the engine-crash cause only produced source after I asked “why?”