AI Disclosure:
I did use AI to assist with the coding of these. I am fairly competent at rust coding, but frankly didn’t have the time to fully do this myself. I did read over the code and it seems sane enough so I just went with it.
Every good project is a solution to a problem. So let’s start with the problem. I have 3 machines on my network that would make for good LLM servers, my home server, my gaming desktop, and my wife’s gaming desktop. One of those machines can be used for LLMS basically 24/7, but is often slower than the other two because its doing a lot more than JUST LLMs, the other two are faster, but sometimes shouldn’t be used because we’re gaming on them.
For the gaming desktops I just made a quick and dirty bash script that runs when the systems are idle that starts the lemond service and then waits for the user to press enter in the konsole window which stops the lemond service again. With that out of the way I now needed a single endpoint to add to my hermes instances that would smartly route llm requests to one of these three machines based on LLM load and whether the LLM server is available or not, but also have a preference since the gaming machines are faster than the server.
Enter LLM-Router (genius name I know) I wrote it with the help of AI in Rust because Rust is what I have the most experience with, so if it did something incredibly stupid I’d be able to tell. Then another problem surfaced; I would like to know what servers are being used, and which ones are available, but I don’t want a dashboard or anything complex, just a system tray would be good enough. So again with the help of AI I modified the server to not only route requests, but also host a /router/status page that shows the status of each tracked server. I then wrote a companion system tray app that checks this page for the servers’ status and displays them when hovered over.
The llm-router binary itself does the system tray as well so you don’t need to run both on the router host. You would just run the router binary on the router host, and the companion app on other machines. You can check them both out at the below links:
screenshot of the system tray app: