Modify AnythingLLM to allow Embed model to stay resident in GPU VRAM

Oct 13, 2025 by talynt.org 131 views

[FEAT]: AnythingLLM only allows the LLM to stay resident in GPU VRAM, with no option for the Embed model

In the realm of Large Language Models (LLMs), AnythingLLM and Ollama are two popular tools that enable developers to create and deploy AI-powered applications. However, a recent issue has been reported by users, where the Embed model fails to stay resident in GPU VRAM, causing a significant delay in user experience. In this article, we will delve into the details of this issue and explore possible solutions to address it.

The problem lies in the fact that AnythingLLM only allows the LLM to stay resident in GPU VRAM, with no option for the Embed model. This means that the Embed model unloads every 3-5 minutes, causing a long delay when someone starts a chat. To mitigate this issue, a curl POST heartbeat was implemented every 2.5 minutes to keep the Embed model loaded. However, this solution is not working as expected, and the Embed model still unloads when a new chat starts.

The current workaround involves implementing a curl POST heartbeat every 2.5 minutes to keep the Embed model loaded. However, this solution has its limitations. From nvidia-smi, it is observed that the heartbeat does load the Embed model, but when a new chat starts, Ollama unloads the heartbeat's Embed model instance and loads a new one for the chat. This behavior is causing significant trouble, as it leads to a delay in user experience.

A subtle behavior has been observed, where another client on the LAN initiating a chat can reuse the loaded Embed model from the previous session. This behavior is causing significant trouble, as it leads to a delay in user experience.

There are several possible solutions to address this issue:

1. Modify AnythingLLM to allow Embed model to stay resident in GPU VRAM

One possible solution is to modify AnythingLLM to allow the Embed model to stay resident in GPU VRAM. This would eliminate the need for the curl POST heartbeat and ensure that the Embed model remains loaded throughout the chat session.

2. Implement a more robust heartbeat mechanism

Another possible solution is to implement a more robust heartbeat mechanism that can keep the Embed model loaded for an extended period. This could involve using a more frequent heartbeat or implementing a more sophisticated mechanism to keep the Embed model loaded.

3. Use a different LLM or Embed model

A third possible solution is to use a different LLM or Embed model that does not have this issue. This could involve switching to a different tool or library that provides a more robust and reliable solution.

In conclusion, the issue of the Embed model failing to stay resident in GPU VRAM is a significant problem that affects user experience. While a curl POST heartbeat has been implemented as a workaround, it is not a reliable solution. There are several possible solutions to address this issue, including modifying AnythingLLM, implementing a more robust heartbeat mechanism, or using a different LLM or Embed model. By exploring these solutions, developers can create more robust and reliable AI-powered applications.

We are using AnythingLLM (latest version) + Ollama (latest version) LLM + Ollama Embed model. We have 2 GPUs with 16GB VRAM each, totaling 32GB. The LLM occupies ~50% of each GPU’s VRAM, while the Embed model uses ~2GB, leaving plenty of free VRAM.

We would like to know if there is a simpler, more reliable solution to keep the Embed model persistently loaded in GPU VRAM. We are open to exploring different solutions and would appreciate any suggestions or recommendations.

We are using AnythingLLM (latest version) + Ollama (latest version) LLM + Ollama Embed model.
We have 2 GPUs with 16GB VRAM each, totaling 32GB.
The LLM occupies ~50% of each GPU’s VRAM, while the Embed model uses ~2GB, leaving plenty of free VRAM.
We implemented a curl POST heartbeat every 2.5 minutes to keep it loaded, but it’s not working as expected.
From nvidia-smi, we see the heartbeat does load the Embed model, but when a new chat starts, Ollama unloads the heartbeat’s Embed model instance and loads a new one for the chat.
However, once a chat is active, the Embed model stays in VRAM, and we’ve noticed that another client on the LAN initiating a chat can reuse the loaded Embed model from the previous session.
Q&A: AnythingLLM and Embed Model Issues =============================================

In our previous article, we discussed the issue of the Embed model failing to stay resident in GPU VRAM when using AnythingLLM and Ollama. We explored possible solutions to address this issue and provided additional information about the problem. In this article, we will answer some frequently asked questions (FAQs) related to this issue.

A: The current workaround involves implementing a curl POST heartbeat every 2.5 minutes to keep the Embed model loaded. However, this solution is not working as expected, and the Embed model still unloads when a new chat starts.

A: When a new chat starts, Ollama unloads the heartbeat's Embed model instance and loads a new one for the chat. This behavior is causing significant trouble, as it leads to a delay in user experience.

A: Yes, you can use a different LLM or Embed model that does not have this issue. This could involve switching to a different tool or library that provides a more robust and reliable solution.

A: Modifying AnythingLLM to allow the Embed model to stay resident in GPU VRAM would require significant changes to the code. This would involve adding a new feature to keep the Embed model loaded throughout the chat session.

A: Keeping the Embed model loaded in GPU VRAM can improve user experience by reducing the delay in chat sessions. It can also improve performance by reducing the time it takes to load the Embed model.

A: Yes, you can use a different heartbeat mechanism to keep the Embed model loaded. This could involve using a more frequent heartbeat or implementing a more sophisticated mechanism to keep the Embed model loaded.

A: To troubleshoot issues related to the Embed model not staying resident in GPU VRAM, you can use tools such as nvidia-smi to monitor the GPU usage and identify any issues related to the Embed model.

A: Yes, you can use a cloud-based service to host your LLM and Embed model. This can provide a more scalable and reliable solution for hosting your LLM and Embed model.

A: To ensure that your LLM and Embed model are secure and compliant with regulations, you can use tools such as encryption and access controls to protect your data. You can also use services such as AWS SageMaker to host your LLM and Embed model in a secure and compliant environment.

In conclusion, the issue of the Embed model failing to stay resident in GPU VRAM is a significant problem that affects user experience. By understanding the current workaround, the reasons for the issue, and the possible solutions, you can take steps to address this issue and improve the performance and reliability of your LLM and Embed model.