In the weeks leading up to our most recent production run, our local LLM agent fleet experienced a series of failures that went unnoticed for days. Looking at the failure ledger, several patterns emerge:
Process Errors in Inference: Multiple entries show process_error
failures during the secretary/consult-classify
and secretary/consult-fields
stages. For example:
ollama/qwen3.8:27b
model encountered a process_error
with a duration of 318,454ms.Timeouts in Agent Evaluation: On September 1st, two consecutive timeouts occurred during hr/agent-evaluation
:
ollama/qwen3.8:27b
and qwen3.6:27b
models, respectively. Each timeout lasted approximately 300 seconds.These failures indicate that while the agents were running unattended, they encountered issues that prevented them from completing their tasks. The long durations and timeouts suggest that the models were either stuck in a loop or encountered resource constraints that were not being addressed.
The root cause of these failures can be attributed to several factors:
Model Instability: The models being used, such as ollama/qwen3.8:27b
and ollama/qwen3.6:27b
, may have inherent instability when running continuously without human oversight. This could be due to memory leaks, inefficient garbage collection, or other performance issues.
Resource Constraints: Running multiple agents simultaneously can strain system resources, leading to timeouts and process errors. The long durations recorded in the failure ledger suggest that the agents were consuming excessive CPU or memory, causing the system to become unresponsive.
Lack of Monitoring: Without proper monitoring in place, these failures went unnoticed for an extended period. This highlights the importance of having real-time alerts and monitoring tools to detect and address issues promptly.
To prevent similar issues in the future, we implemented several mechanical guards:
Automated Alerts: We set up automated alerts to notify the engineering team immediately when a process_error
or timeout occurs. This ensures that any issues are addressed promptly, minimizing downtime and preventing prolonged failures.
Resource Monitoring: We implemented resource monitoring to track CPU and memory usage of the agents. If an agent exceeds predefined thresholds, it is automatically restarted or reconfigured to use fewer resources.
Model Health Checks: Regular health checks are now performed on the models to ensure they are running optimally. This includes checking for memory leaks, optimizing garbage collection, and ensuring that the models are not consuming excessive resources.
Failover Mechanisms: We introduced failover mechanisms to switch to a backup model or agent if the primary one fails. This ensures that the agent fleet can continue operating even if individual components encounter issues.
The key lesson from this experience is the importance of proactive monitoring and maintenance in unattended agent fleets. While running agents unattended can increase efficiency and productivity, it also introduces risks that must be managed carefully. By implementing automated alerts, resource monitoring, regular health checks, and failover mechanisms, we can ensure that our agent fleet remains stable and reliable, even when running for extended periods.
In conclusion, the 58-day unattended run highlighted the silent failures that can occur in local LLM agent fleets. By addressing these issues with the mechanical guards we put in place, we have improved the resilience and reliability of our system. Moving forward, we will continue to refine our monitoring and maintenance strategies to ensure that our agent fleet operates smoothly and efficiently.