Before you rent a GPU, choose the part of your application you want an open model to handle. That decision will save you a lot of aimless infrastructure work.
Get a support assistant. It might search a knowledge base, rank the matching pages, and ask a large language model to write the answer.
You could move its embedding model to an open alternative while leaving answer generation with your existing provider. That’s a manageable first project.
Now you have something specific to evaluate: does search still find the right pages, how long does it take, and what will it cost to run? Those questions give you a much better starting point than a list of GPUs.
I’ll use Superlinked Inference Engine (SIE) for the deployment examples below. You can begin locally, use a managed endpoint, or run the engine in your own cloud.
The choice depends on how much of the service you need to operate yourself.
START WITH A SMALL SCOPE
Choose one task, collect examples from the application, and compare the open model with what you use today. Once the results are good enough, decide where the model should run. You don’t need to move the whole application at once.
For the support assistant, embeddings are one possible starting point. Reranking retrieved pages could be another. Both give you a fairly contained part of the application to test.
Other applications might need field extraction from documents or a small language model for a narrow generation task.
Pick a job where you can inspect the result and explain what a bad result looks like.
Be specific about the reason for moving it. Perhaps the data has to stay in your environment.
Perhaps one frequently called model accounts for a large share of your bill. Or you need more control over the model version.
If your main requirement is the strongest available reasoning on difficult requests, keep a frontier API in the design.
The small open LLMs targeted by SIE can handle suitable tasks around it. Whether they can handle yours needs testing.
The SIE model catalog can help you find candidates. Start with a short list. Trying every available model makes the evaluation harder to finish.
Save a set of representative inputs and the results from your current system before changing anything. Include the difficult cases.
For the support assistant, that might mean vague questions, product names with typos, and questions that the knowledge base cannot answer.
For an embedding model, compare which pages appear near the top of the search results.
A request that returns a vector successfully tells you that the endpoint works; it says nothing about whether the assistant found the right instructions.
For extraction, check the actual field values. For generation, check whether the answer follows the task and stays within the information you supplied.
Write down which mistakes you can tolerate and which would rule the model out.
Set a response-time budget as well. If search already uses most of the assistant’s acceptable wait time, a slower embedding model could make an otherwise good replacement impractical.
You can do an initial quality check on a local machine. Save capacity testing for the hardware and deployment you expect to use.
Using an open model and running your own hardware are separate decisions. A managed open-model endpoint lets you use the model while the provider handles its servers.
If you don’t yet know how much traffic you’ll have, a managed endpoint gives you room to find out.
With self-hosting, you pay for the capacity you keep available, including the hours when almost nobody uses it.
Decide how much operational work belongs with your team.
SIE Cloud is the managed route for the same engine. If you need self-hosting, SIE also has Docker and Kubernetes deployments.
For cost planning, use your own traffic estimates. Superlinked’s self-hosting discussion includes worked break-even examples, but a request count alone won’t tell you whether a GPU is cheaper.
Input size, utilization, and the time spent operating the service all affect the answer.
For a local SIE check on Apple Silicon, the quickstart uses:
pip install "sie-server[local]"sie-server serve
This starts a server at http://localhost:8080. SIE's local NVIDIA instructions add --device cuda to the serve command.
If you already use the OpenAI Python client, you can connect it to SIE’s OpenAI-compatible API. For a client running on the same machine as the local server:
from openai import OpenAI
client = OpenAI( api_key="not-needed", base_url="http://localhost:8080/v1",)
The migration guide also covers selecting a replacement model. Its embedding example uses intfloat/e5-base-v2 in place of text-embedding-3-small.
Read the replacement model’s input requirements before evaluating it.
For SIE Cloud, the endpoint is https://api.superlinked.com/v1; follow the managed service's authentication instructions.
The dummy key in the local example isn’t a Cloud credential.
SIE’s Docker example starts the GPU server with a persistent model cache:
docker run --gpus all -p 8080:8080 \ -v sie-hf-cache:/app/.cache/huggingface \ ghcr.io/superlinked/sie-server:latest-cuda12-default
The cache matters when you recreate the container: downloaded weights can be reused. The model still needs time to load into memory.
Use the deployment documentation to check hardware requirements.
This command starts a server; the access controls and recovery arrangements still belong in your deployment plan.
Back to the support assistant. You have an open embedding model answering requests, and the client connects correctly.
There is still an entire knowledge base indexed with the previous model.
Changing the model name in the query code won’t migrate that index. Embeddings from different models use different vector spaces.
A query embedded by the new model should be searched against documents embedded by that model too.
Create a separate index from freshly embedded documents, then run the saved questions against it.
Compare the retrieved pages with the old system’s results before directing users to it.
During the rollout, treat the query model and its document index as a pair. If you switch one, switch the other with it.
Keep the previous pair available until you’re satisfied with the replacement.
BUDGET FOR RE-EMBEDDING
Include the time and compute needed to rebuild the corpus when estimating the migration.
For a large knowledge base, that work may take longer than setting up the model server.
On a managed service, your team still needs to handle failed requests and provider limits in the application.
On a self-hosted server, it also needs to notice when the worker stops and know how to recover it.
For SIE, pay particular attention to model . Models load on demand, and the engine evicts the least recently used model when GPU memory fills up.
A request can therefore be slow even when the server process is healthy.
Try the request sequence your application will create. If it alternates between embedding, reranking, and extraction models, test that combination.
Running each model separately won’t reveal delays caused by repeatedly them.
Name the person who will respond when it fails. Decide how long a user should wait and what the application should return when inference cannot finish.
A request left hanging gives the user very little to work with.
If you need multiple workers for availability or capacity, SIE provides Kubernetes deployment options and Terraform modules for GCP, AWS, and Azure.
Account for startup time when choosing scaling settings. A worker that has to download and load a model cannot absorb a traffic spike immediately.
Test how long it takes in your environment before relying on automatic scaling to cover a sudden burst.
For the support assistant, I’d write down a short release plan with these items:
Give a limited group access first and compare real behavior with the evaluation.
If search results deteriorate or waiting times grow, you’ll have a smaller rollout to investigate and a working configuration to return to.
Once that workload is stable, use what you learned to decide whether another task should move.
You might bring reranking into the same setup, or discover that the existing API is still a good fit for the remaining work.
The SIE quickstart covers the local setup. Use it to run your saved examples, then keep the results alongside the configuration.
You’ll need that baseline when you change the model later.
Want to Run Open-Source AI in Production? Start Here was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.