Investigating Speculative Decoding

Oct 13, 2025 by talynt.org 126 views

=====================================================

Bug Description

Speculative Decoding is a feature in Llama-box that enables the model to generate responses based on the context of the conversation. However, in this case, the Llama-box instance crashes on the first chat request, resulting in an error message.

Steps to Reproduce

To reproduce this issue, follow these steps:

Step 1: Create a Llama 3.3 70B Q4_K_M Model Instance

Create a Llama 3.3 70B Q4_K_M model instance with the Llama 3.2 1B Q8 model as a draft model and full offloading of the draft model. Use the following command:

arguments  : /var/lib/gpustack/bin/llama-box_v0.0.125 --host 0.0.0.0 --embeddings --gpu-layers 81 --parallel 4 --ctx-size 8192 --port 40405 --model /models/gguf/Llama-3.3-70B-Instruct-Q4_K_M.gguf --alias llama-3.3-70b-instruct-q4km-sd --no-mmap --no-warmup --ctx-size 16384 --flash-attn --cache-type-k q8_0 --cache-type-v q8_0 --jinja -md /models/gguf/Llama-3.2-1B-Instruct-Q8_0.gguf -ngld 99 --parallel 2

Step 2: Run the Llama-box Instance

Run the Llama-box instance using the command above.

Step 3: Try a Chat Completion

Try a chat completion via the playground.

Result

When trying a chat completion via the playground, the following message appears and the Llama-box instance crashes:

0.46.030.642 E llama_get_logits_ith: invalid logits id 0, reason: no logits (Error Code 139)

Expected Behavior

The expected behavior is that the Llama-box instance should not crash and should be able to handle chat completions without any issues.

Environment

The environment used to reproduce this issue is:

GPUStack version: Server b64bb40 UI e04874d
OS: Debian 12, Docker 28.0.1
GPU: Nvidia RTX A6000, Driver Version: 570.124.06
CUDA Version: 12.8 (the error also happened with Cuda 12.6 and slightly older driver)

Additional Context

The same configuration works out of the box using llama.cpp llama-server:

./llama-server -m Llama-3.3-70B-Instruct-Q4_K_M.gguf -c 16384 -fa -ctk q8_0 -ctv q8_0 -np 2 --host 0.0.0.0 --jinja -md Llama-3.2-1B-Instruct-Q8_0.gguf -ngl 99 -ngld 99

Possible Causes

The possible causes of this issue are:

Speculative Decoding: The Speculative Decoding feature may be causing the Llama-box instance to crash.
Draft Model: The draft model may be causing the issue.
GPU Configuration: The GPU configuration may be causing the issue.

Conclusion

In conclusion, the Llama-box instance crashes on the first chat request due to the Speculative Decoding feature. The same configuration works out of the box using llama.cpp llama-server. The possible causes of this issue are Speculative Decoding, draft model, and GPU configuration.

Future Work

Future work includes:

Investigating Speculative Decoding: Investigate the Speculative Decoding feature to determine if it is causing the issue.
Testing Draft Model: Test the draft model to determine if it is causing the issue.
Optimizing GPU Configuration: Optimize the GPU configuration to determine if it is causing the issue.

References

Note: The above article is a rewritten version of the provided content, with additional information and formatting to make it more readable and SEO-friendly.

=====================================================

Frequently Asked Questions

Q: What is Speculative Decoding?

A: Speculative Decoding is a feature in Llama-box that enables the model to generate responses based on the context of the conversation.

Q: Why is the Llama-box instance crashing?

A: The Llama-box instance is crashing due to an error in the Speculative Decoding feature.

Q: What is the error message?

A: The error message is 0.46.030.642 E llama_get_logits_ith: invalid logits id 0, reason: no logits (Error Code 139).

Q: What are the possible causes of this issue?

A: The possible causes of this issue are:

Speculative Decoding: The Speculative Decoding feature may be causing the Llama-box instance to crash.
Draft Model: The draft model may be causing the issue.
GPU Configuration: The GPU configuration may be causing the issue.

Q: How can I reproduce this issue?

A: To reproduce this issue, follow these steps:

Create a Llama 3.3 70B Q4_K_M model instance with the Llama 3.2 1B Q8 model as a draft model and full offloading of the draft model.
Run the Llama-box instance using the command above.
Try a chat completion via the playground.

Q: What is the expected behavior?

A: The expected behavior is that the Llama-box instance should not crash and should be able to handle chat completions without any issues.

Q: What is the environment used to reproduce this issue?

A: The environment used to reproduce this issue is:

GPUStack version: Server b64bb40 UI e04874d
OS: Debian 12, Docker 28.0.1
GPU: Nvidia RTX A6000, Driver Version: 570.124.06
CUDA Version: 12.8 (the error also happened with Cuda 12.6 and slightly older driver)

Q: Can I use llama.cpp llama-server instead of Llama-box?

A: Yes, you can use llama.cpp llama-server instead of Llama-box. The same configuration works out of the box using llama.cpp llama-server.

Q: What is the next step to resolve this issue?

A: The next step to resolve this issue is to investigate the Speculative Decoding feature, test the draft model, and optimize the GPU configuration.

Additional Resources

Conclusion

In conclusion, the Llama-box instance crashes on the first chat request due to the Speculative Decoding feature. The possible causes of this issue are Speculative Decoding, draft model, and GPU configuration. The next step to resolve this issue is to investigate the Speculative Decoding feature, test the draft model, and optimize the GPU configuration.