Originally published on tamiz.pro.
Imagine it’s 2 AM. You’re staring at a gnarly segmentation fault in a C++ module that processes financial data. You reach for your AI coding assistant, but your hands hesitate. That proprietary cloud service is exactly where you do NOT want your proprietary algorithm or sensitive data schema to be. This is the "Friend's Problem": the growing anxiety among engineers that using AI for development is becoming a trade-off between convenience and confidentiality.
The solution isn't to abandon AI. It's to localize it. By leveraging open-weight models (like Meta's Llama 3, Mistral, or TinyLlama) and inference engines like Ollama, we can build high-fidelity developer tools that run entirely on the local machine. No API keys, no data exfiltration, just pure, offline power.
In this guide, we will move beyond simple chatbots. We will architect a "Privacy-First Developer Tool"—a CLI that acts as a local code reviewer and document generator. We will cover model selection, the architecture of local inference, and how to wire up a secure, deterministic application that respects your data's privacy.
The "Friend's Problem" is a colloquial term in the enterprise security community for the dilemma of trusting a third party (your "friend" or vendor) with sensitive intellectual property. In software engineering, this manifests as the risk of sending proprietary code to LLM APIs.
While cloud-hosted LLMs offer state-of-the-art capabilities, they introduce several risks:
Open-weight models solve this by allowing the model files themselves to reside on your disk. The most accessible way to manage these models today is via Ollama, a runtime that simplifies the download, management, and serving of LLMs.
To build this tool, you need a machine with a decent amount of RAM. LLMs are memory-hungry.
pip`` brew/ docker)
If you haven't already, install Ollama. It acts as the backend service that loads models into memory and provides a REST API.
Verify it's running:
curl http://localhost:11434/api/tags
Not all open-weight models are created equal. For a developer tool, we prioritize reasoning and code generation over general chat capabilities.
Recommendation: Llama 3 8B or Mistral 7B
For this tutorial, we will use Llama 3 8B.
Before coding, let's look at the data flow. Unlike a cloud API, our architecture is closed-loop.
graph LR
A[User Input: Code/Query] --> B(Local Pre-processing: Sanitize & Chunk)
B --> C(Ollama REST API)
C --> D[LLA Inference Engine on Local GPU/CPU]
D --> E[Streaming Response]
E --> F(Post-processing: Format & Highlight)
F --> G[CLI Output to Terminal]
Key components:
First, we need to pull the model. This step happens once.
ollama pull llama3:8b
You can verify it's available:
ollama list
Now, let's test the model manually to ensure it's responsive. Ollama provides a simple CLI:
curl http://localhost:11434/api/generate -d '{
"model": "llama3:8b",
"prompt": "Explain the difference between a stack and a queue in 20 words.",
"stream": false
}'
If this returns a JSON response with valid text, your backend is ready. Note the stream: false flag. We will implement streaming in Python for a better UX, but for testing, synchronous is easier.
We will build a lightweight Python class to handle the communication with Ollama. We'll use requests for the HTTP calls.
Create a file named local_ai_tool.py.
import requests
import json
import time
LOCAL_AI_TOOL_CONFIG = {
"BASE_URL": "http://localhost:11434",
"MODEL": "llama3:8b",
"TEMPERATURE": 0.1, # Low temperature for deterministic code generation
"SYSTEM_PROMPT": "You are a Senior Security Engineer. You are concise and helpful. When asked to review code, focus on vulnerabilities and edge cases."
}
class OllamaClient:
def __init__(self, base_url: str, model: str, temperature: float = 0.1):
self.base_url = base_url
self.model = model
self.temperature = temperature
def generate(self, prompt: str, system_prompt: str = None) -> str:
"""
Synchronous generation. Blocks until complete.
Useful for scripts and automated tests.
"""
endpoint = f"{self.base_url}/api/generate"
payload = {
"model": self.model,
"prompt": prompt,
"system": system_prompt or LOCAL_AI_TOOL_CONFIG["SYSTEM_PROMPT"],
"temperature": self.temperature,
"stream": False
}
try:
response = requests.post(endpoint, json=payload, timeout=60)
response.raise_for_status()
result = response.json()
return result.get("response", "")
except requests.exceptions.ConnectionError:
raise Exception("Could not connect to Ollama. Is the service running?")
except json.JSONDecodeError:
raise Exception("Invalid JSON response from Ollama.")
def generate_streaming(self, prompt: str, system_prompt: str = None):
"""
Streaming generation. Yields tokens as they are generated.
This provides immediate feedback to the user.
"""
endpoint = f"{self.base_url}/api/generate"
payload = {
"model": self.model,
"prompt": prompt,
"system": system_prompt or LOCAL_AI_TOOL_CONFIG["SYSTEM_PROMPT"],
"temperature": self.temperature,
"stream": True
}
with requests.post(endpoint, json=payload, stream=True, timeout=120) as response:
for line in response.iter_lines():
if line:
json_line = json.loads(line)
yield json_line.get("response", "")
client = OllamaClient(
base_url=LOCAL_AI_TOOL_CONFIG["BASE_URL"],
model=LOCAL_AI_TOOL_CONFIG["MODEL"],
temperature=LOCAL_AI_TOOL_CONFIG["TEMPERATURE"]
)
LLM generation is sequential. Waiting for the entire block to finish (synchronous) can take 10-20 seconds. Streaming allows the user to see the "stream of consciousness" as it happens, which makes the tool feel much more responsive and allows you to interrupt (Ctrl+C) if it's going off the rails.
A raw prompt is not enough. We need to structure the input so the model knows what to do. For a developer tool, we often want to feed in a file's content.
Let's create a utility to read a file and wrap it in a specific prompt structure. We also need to handle the "Privacy" aspect. Even locally, if this tool is ever shared or if the model has been fine-tuned on public data, we want to minimize the risk of accidental leakage of secrets (API keys, DB passwords).
import re
def sanitize_code(code: str) -> str:
"""
Basic regex-based sanitizer to mask common secrets.
Note: This is NOT a comprehensive DLP tool. It's a safety net.
"""
code = re.sub(r"AKIA[0-9A-Z]{16}", "[AWS_KEY_MASKED]", code)
code = re.sub(r"api_key\s*[:=]\s*['\"]?[a-zA-Z0-9]{20,}['\"]?", "api_key: '[KEY_MASKED]'", code)
code = re.sub(r"(password|pwd)\s*[:=]\s*['\"]?[^'\"]{5,}['\"]?", r"\1: '[PWD_MASKED]'", code)
return code
def format_review_prompt(file_content: str, filename: str) -> str:
"""
Construct a structured prompt for code review.
"""
sanitized_content = sanitize_code(file_content)
prompt_template = """
### Instructions:
You are reviewing the following file: `{filename}`.
### Requirements:
1. Identify security vulnerabilities (SQL injection, XSS, hardcoded secrets).
2. Identify performance bottlenecks (N+1 queries, unnecessary allocations).
3. Provide specific code snippets to fix the issues.
4. Do NOT rewrite the entire file unless specifically asked. Just provide the diffs or specific function replacements.
### File Content:
```
python
{sanitized_content}
```
### Response Format:
- **Security Risks:**
- **Performance Issues:**
- **Recommended Fixes:**
"""
return prompt_template.format(filename=filename, sanitized_content=sanitized_content)
Now we tie it all together. We will use click (or just argparse to keep dependencies minimal, but click is cleaner). Let's stick to standard library argparse for zero external dependencies in the CLI layer.
import argparse
import sys
import os
def print_stream(stream):
"""Helper to print streaming tokens to stdout with newline control."""
for token in stream:
print(token, end='', flush=True)
print() # Final newline
def main():
parser = argparse.ArgumentParser(description="Local Privacy-First Code Reviewer")
parser.add_argument("file", type=str, help="Path to the Python file to review")
parser.add_argument("-q", "--question", type=str, help="Optional specific question to ask the model about the file")
parser.add_argument("-y", "--yes", action='store_true', help="Skip confirmation before running")
args = parser.parse_args()
if not os.path.exists(args.file):
print(f"Error: File '{args.file}' not found.", file=sys.stderr)
sys.exit(1)
try:
with open(args.file, 'r') as f:
content = f.read()
except Exception as e:
print(f"Error reading file: {e}", file=sys.stderr)
sys.exit(1)
prompt = format_review_prompt(content, os.path.basename(args.file))
if args.question:
prompt += f"\n\n### Additional Context:\n{args.question}"
if not args.yes:
print("\n--- Prompt Ready ---")
print(prompt[:200] + "..." if len(prompt) > 200 else prompt)
print("\nRunning local inference... This may take a moment. [Ctrl+C to cancel]")
input("Press Enter to start, or Ctrl+C to abort: ")
print("\n# Analysis Report \n")
try:
stream = client.generate_streaming(prompt)
print_stream(stream)
except KeyboardInterrupt:
print("\n[Cancelled by user]")
sys.exit(0)
except Exception as e:
print(f"Error during inference: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
Save the script. Make it executable.
chmod +x local_ai_tool.py
python3 local_ai_tool.py my_secure_script.py
The tool will read my_secure_script.py, sanitize it, send it to the local Llama 3 instance, and stream the security review back to your terminal. All of this happened on your machine.
If your machine is struggling (high CPU usage, slow generation), you are likely using a full-precision model on a CPU. Here is how to optimize.
Ollama typically downloads quantized versions (Q4_0) by default, which is a good balance. However, you can request higher precision for better accuracy if you have the VRAM.
ollama pull llama3:8b --precision f16
Note: This doubles the memory footprint. Use only if you have an NVIDIA 3090/4090 or high-end Apple Silicon (M1/M2/M3 Max).
The num_ctx parameter in Ollama controls how much memory is reserved for the context window. The default is often small (2048 tokens).
To allow the model to review large files, you need to increase this.
Modelfile:
FROM llama3:8b
PARAMETER num_ctx 8192
ollama create my-llama-8k -f Modelfile
"my-llama-8k" instead of "llama3:8b".
Just because the model is local doesn't mean the data is safe. If you are using this on a shared workstation, anyone with admin rights can read the memory or the model files.
/dev/shm or memory dumps.
Open-weight models are less sophisticated than GPT-4 in detecting subtle logical errors. They are great at syntax and obvious security patterns, but they might miss a race condition in a multi-threaded app.
We set temperature to 0.1 in our config. This is crucial for code.
Open-weight models are static files. They do not update automatically. You will eventually need to re-download them to get the latest improvements. Keep your Ollama setup up to date.
Q: Can I use this on a machine with no GPU?
A: Yes. Ollama and Llama 3 8B run on CPU-only machines, but expect generation speeds of 2-5 tokens per second. It is usable for asynchronous tasks (e.g., background code review) but slow for interactive chat. For interactive use, a GPU (NVIDIA or Apple Silicon) is highly recommended.
Q: How do I prevent the model from leaking sensitive data into the output?
A: The prompt engineering in Step 3 uses a "Privacy-First" approach by sanitizing the input. However, the model can still repeat data it sees. To be safe, always use the sanitize_code function on the input and consider adding a post-processing regex filter on the output to catch any accidental PII that the model might have hallucinated or repeated from the context.
Q: Is Ollama the only way to do this?
A: No. You can also use llama.cpp directly (for maximum performance) or vLLM (for high-throughput serving). Ollama is chosen here for its simplicity and developer-friendly API. For a production-grade developer tool, you might want to wrap vLLM for better concurrency handling.
You have now built a privacy-first developer tool that empowers you to use the power of LLMs without sacrificing your intellectual property or data confidentiality. By leveraging open-weight models like Llama 3 and the simplicity of Ollama, you can integrate AI into your workflow in a way that respects the boundaries of your codebase.
This approach is not just about security; it's about sovereignty. You own your tools, you control your data, and you run your workflow on your own terms. As open models continue to improve, this "local-first" strategy will likely become the standard for enterprise and security-conscious developers.
For more insights on local AI architecture, check out our previous breakdown on Vector Databases for Local RAG and how to Optimize LLM Inference on Edge Devices.