In this article, you will learn how to build a side-by-side benchmark of local and cloud LLM classifiers using Scikit-Ollama and a Groq-backed Scikit-LLM pipeline.
Topics we will cover include:
- How to set up Ollama, Scikit-Ollama, and Scikit-LLM, and how to create a small sentiment dataset for testing.
- How to write a reusable helper function that measures the execution time and accuracy of a zero-shot classifier.
- How to run the same benchmark locally and through the Groq API, and what the tradeoffs are in latency, cost, setup, and privacy.
Introduction #
Looking for a practical, side-by-side comparison of two LLM solutions (local vs. cloud) in the same application scenario? This article presents a head-to-head benchmark of two different solutions:
- A local Ollama -backed machine learning pipeline constructed with the aid ofScikit-Ollama .
- A cloud LLM equivalent, in this case a Groq-backed Scikit-LLM pipeline with the same purpose as the first, local LLM-based scenario.
For the target tasks shown, the comparison will quantify the real tradeoffs across dimensions like speed, accuracy, and real (or potential) cost.
Environment Setup and Data Creation #
The first step of the process is to install the required Python libraries, mainly scikit-llm and scikit-ollama. If you want to run the code in a notebook, you’ll also need to install the Ollama engine to download and use open-source LLMs “locally”. If you are using an IDE on your local machine, make sure to have Ollama installed separately from its website and pull (download) the local llama3 model. Either way, the code includes the instructions to start the Ollama server in the background and pull the model.
import subprocess
import time
!pip install -q scikit-llm scikit-ollama
!apt-get update -qq && apt-get install -y -qq zstd
!curl -fsSL https://ollama.com/install.sh | sh
print("Starting Ollama server...")
process = subprocess.Popen(["ollama", "serve"])
time.sleep(5)
print("Down Llama 3...")
!ollama pull llama3
print("Environment is ready!")
1234567891011121314151617181920
import subprocessimport time # 1. Installing wrapper libraries!pip install -q scikit-llm scikit-ollama # 2. Installing Ollama (not needed if you use a local IDE and already have it installed)# and starting background server!apt-get update -qq && apt-get install -y -qq zstd!curl -fsSL https://ollama.com/install.sh | sh print("Starting Ollama server...")process = subprocess.Popen(["ollama", "serve"])time.sleep(5) # 3. Pulling Llama3 modelprint("Down Llama 3...")!ollama pull llama3 print("Environment is ready!")
If the setup process went well, we are ready for the next step: creating the dataset and defining a helper benchmark function. This stage is basically broken down as follows:
First, let’s create a tiny, dummy dataset with texts and their associated ground-truth sentiment labels (y_true):
import time
import numpy as np
from sklearn.metrics import accuracy_score
texts = [
"I absolutely loved the new update, everything is so fast!",
"The customer service was terrible and unhelpful.",
"Works exactly as described, highly recommend.",
"My screen broke after two days of use.",
"It's okay, nothing special but it does the job."
]
y_true = ["Positive", "Negative", "Positive", "Negative", "Positive"]
candidate_labels = ["Positive", "Negative"]
1234567891011121314
import timeimport numpy as npfrom sklearn.metrics import accuracy_score texts = [ "I absolutely loved the new update, everything is so fast!", "The customer service was terrible and unhelpful.", "Works exactly as described, highly recommend.", "My screen broke after two days of use.", "It's okay, nothing special but it does the job."] y_true = ["Positive", "Negative", "Positive", "Negative", "Positive"]candidate_labels = ["Positive", "Negative"]
Second, to make our code reusable and easily extendable to new comparisons under a centralized testing logic, we will define a “universal” helper function called benchmark_estimator. This function is responsible for handling the pipeline’s fit() and predict() methods. As you will observe later, both the local and cloud estimators we instantiate will be passed through the exact same function, thereby measuring aspects like accuracy and execution time fairly.
def benchmark_estimator(name, clf, texts, y_true, candidate_labels, cost_note=""):
print(f"--- Running Benchmark: {name} ---")
start_time = time.perf_counter()
clf.fit(None, candidate_labels)
predictions = clf.predict(texts)
elapsed_time = time.perf_counter() - start_time
accuracy = accuracy_score(y_true, predictions)
print(f"Time Elapsed : {elapsed_time:.2f} seconds")
print(f"Accuracy : {accuracy * 100:.1f}%")
print(f"Predictions : {list(predictions)}")
print(f"Cost : {cost_note}\n")
1234567891011121314
def benchmark_estimator(name, clf, texts, y_true, candidate_labels, cost_note=""): print(f"--- Running Benchmark: {name} ---") start_time = time.perf_counter() clf.fit(None, candidate_labels) predictions = clf.predict(texts) elapsed_time = time.perf_counter() - start_time accuracy = accuracy_score(y_true, predictions) print(f"Time Elapsed : {elapsed_time:.2f} seconds") print(f"Accuracy : {accuracy * 100:.1f}%") print(f"Predictions : {list(predictions)}") print(f"Cost : {cost_note}\n")
Scenario 1: Local Zero-Shot Classification with Scikit-Ollama #
For the first run of our benchmark, we initialize a ZeroShotOllamaClassifier from Scikit-Ollama, targeting our previously downloaded Llama3 model.
After that, we run the benchmark. How? Simpler than it sounds, as we already have our helper function to do so! We just need to carefully pass the right arguments to the function: a descriptive name for this benchmark estimator, the zero-shot classifier instance we just defined, the data, the ground-truth labels, and the existing class labels for the problem, stored in candidate_labels.
In sum, we are testing the local side of our zero-shot classification pipeline, proving how the model can classify texts entirely on a local machine (or Google Colab session, for that matter) without depending on external servers, guaranteeing absolute data privacy and avoiding API costs.
from skollama.models.ollama.classification.zero_shot import ZeroShotOllamaClassifier
local_clf = ZeroShotOllamaClassifier(model="llama3")
benchmark_estimator(
name="Local: scikit-ollama (Llama 3)",
clf=local_clf,
texts=texts,
y_true=y_true,
candidate_labels=candidate_labels,
cost_note="$0.00 (Local Hardware)"
)
1234567891011121314
from skollama.models.ollama.classification.zero_shot import ZeroShotOllamaClassifier # Initializing the local estimatorlocal_clf = ZeroShotOllamaClassifier(model="llama3") # Executing Local Benchmarkbenchmark_estimator( name="Local: scikit-ollama (Llama 3)", clf=local_clf, texts=texts, y_true=y_true, candidate_labels=candidate_labels, cost_note="$0.00 (Local Hardware)")
Results:
Time Elapsed : 80.71 seconds
Accuracy : 80.0%
Predictions : [np.str_('Positive'), np.str_('Negative'), np.str_('Positive'), np.str_('Negative'), np.str_('Negative')]
Cost : $0.00 (Local Hardware)
1234
Time Elapsed : 80.71 secondsAccuracy : 80.0%Predictions : [np.str_('Positive'), np.str_('Negative'), np.str_('Positive'), np.str_('Negative'), np.str_('Negative')]Cost : $0.00 (Local Hardware)
Cost is inherently $0, as we are running free, open-source, local models with Ollama. While the model scored four out of five (80%) correctly, be aware that for a more realistic evaluation, we would use a larger dataset than just five examples. We kept the dataset to a minimum size to enable smooth, accessible running for anyone, but feel free to add extra data examples (with their associated true labels) if needed.
Scenario 2: Cloud-Backed Zero-Shot Classification with Scikit-LLM and Groq #
To enable the comparison, we will now define — aided by Scikit-LLM — another zero-shot classifier. For this, we use SKLLMConfig.set_gpt_url() to route Scikit-LLM’s native ZeroShotGPTClassifier to a compatible cloud endpoint via custom URL prefixes. Using free cloud models like Groq’s, this approach avoids account quota locks and keeps the benchmark completely free.
You may need to install Scikit-LLM first:
!pip install -q --upgrade scikit-llm
1
!pip install -q --upgrade scikit-llm
Also, be sure to get an API key from the Groq website and add it to a secret variable (in Google Colab) called GROQ_API_KEY.
Then use the following code to execute the second “actor” of our benchmark demonstration:
from google.colab import userdata
from skllm.config import SKLLMConfig
from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier
try:
GROQ_API_KEY = userdata.get('GROQ_API_KEY')
SKLLMConfig.set_openai_key(GROQ_API_KEY)
SKLLMConfig.set_gpt_url("https://api.groq.com/openai/v1/")
cloud_clf = ZeroShotGPTClassifier(model="custom_url::openai/gpt-oss-20b")
benchmark_estimator(
name="Cloud: scikit-llm via Groq API (gpt-oss-20b)",
clf=cloud_clf,
texts=texts,
y_true=y_true,
candidate_labels=candidate_labels,
cost_note="$0.00 (Free Cloud Tier)"
)
except userdata.SecretNotFoundError:
print("Please add a free 'GROQ_API_KEY' (from console.groq.com) to your Google Colab Secrets.")
except Exception as e:
print(f"An error occurred during the API call: {e}")
1234567891011121314151617181920212223242526272829
from google.colab import userdatafrom skllm.config import SKLLMConfigfrom skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier try: # 1. Fetching free Groq API key from Colab Secrets GROQ_API_KEY = userdata.get('GROQ_API_KEY') # 2. Configuring skllm to route traffic to Groq's custom URL instead of OpenAI SKLLMConfig.set_openai_key(GROQ_API_KEY) SKLLMConfig.set_gpt_url("https://api.groq.com/openai/v1/") # 3. Initializing the true scikit-llm classifier using the active model # The "custom_url::" prefix tells scikit-llm to use the URL provided above cloud_clf = ZeroShotGPTClassifier(model="custom_url::openai/gpt-oss-20b") # Executing Cloud Benchmark using the generic helper benchmark_estimator( name="Cloud: scikit-llm via Groq API (gpt-oss-20b)", clf=cloud_clf, texts=texts, y_true=y_true, candidate_labels=candidate_labels, cost_note="$0.00 (Free Cloud Tier)" )except userdata.SecretNotFoundError: print("Please add a free 'GROQ_API_KEY' (from console.groq.com) to your Google Colab Secrets.")except Exception as e: print(f"An error occurred during the API call: {e}")
Results:
Time Elapsed : 2.02 seconds
Accuracy : 100.0%
Predictions : [np.str_('Positive'), np.str_('Negative'), np.str_('Positive'), np.str_('Negative'), np.str_('Positive')]
Cost : $0.00 (Free Cloud Tier)
1234
Time Elapsed : 2.02 secondsAccuracy : 100.0%Predictions : [np.str_('Positive'), np.str_('Negative'), np.str_('Positive'), np.str_('Negative'), np.str_('Positive')]Cost : $0.00 (Free Cloud Tier)
Obviously, once again, the cost is simply a static output of $0, as the Groq API is also free to use as of writing. While the experiment shown here is small for the sake of space, and without attempting to compare accuracy on such a small dataset, we can draw some conclusions that apply if you extend your experiments:
- Latency is mainly limited by GPU VRAM and token generation speed in the local, Ollama-based solution. Meanwhile, it is near-instantaneous in the cloud solution due to its dedicated inference engines.
- Both options we used are free, but if you switch to a different cloud provider, like OpenAI, small costs per inference call may apply.
- The setup overhead is also lower in the case of the cloud solution.
- The greatest advantage of Ollama’s local models is the total data privacy, with potentially sensitive data never leaving local memory.
Wrapping Up #
This article walked you through the process of building a fair, standardized benchmark between local inference and cloud inference for LLM-driven pipelines, illustrating it in zero-shot classification scenarios. Both approaches, making use of Scikit-Ollama and Scikit-LLM respectively, adhere to scikit-learn’s classic .fit() and .predict() methods used in traditional machine learning pipelines.