How to Self-Host a Validated AI Coding Assistant with NVIDIA NeMo Guardrails

NVIDIA has published a tutorial for self-hosting a validated AI coding assistant using StarCoder2-7B NIM and NeMo Guardrails, enabling code completions on local GPUs with policy enforcement, hallucinated-package detection, and audit traceability. The architecture keeps the model as a proposal engine while policy, dependency verification, and metrics live outside the model in trusted systems. The tutorial requires an NGC API key, a supported NVIDIA GPU with at least 24 GB memory, Docker with the NVIDIA Container Toolkit, Python 3.10+, and a Git repo.

Deploying an AI coding assistant in a regulated, sovereign, or source-sensitive environment, often comes with challenges. Three common issues are: the source cannot leave the network, the assistant occasionally invents package names that introduce supply-chain risk, and there is no audit trail when a generated change ships a defect. This tutorial walks you through how to self-host a validated coding assistant on NVIDIA infrastructure that solves all three of these issues. By the end, you’ll have a StarCoder2-7B NIM https://build.nvidia.com/bigcode/starcoder2-7b/deploy endpoint serving code completions from your own GPUs, an NVIDIA NeMo Guardrails https://github.com/NVIDIA/NeMo-Guardrails policy in front of it that refuses requests for files you mark as human-only, a CI verification stage that catches hallucinated packages before review, commit-level traceability, and a minimal metrics loop that tells you whether AI-assisted patches are improving or hurting your defect rate. Tutorial prerequisites and notes To follow along with the tutorial, you’ll need: - An NGC API key - A supported NVIDIA GPU with at least 24 GB of memory for example, NVIDIA A10, L4, L40S, or A100 - Docker with the NVIDIA Container Toolkit https://github.com/NVIDIA/nvidia-container-toolkit - Python 3.10+ - A Git repo you can experiment against StarCoder2-7B runs in BF16. NVIDIA H100 and H200 GPUs provide the certified, highest-throughput profile but are not required for a pilot. Every artifact in this tutorial is shown inline and is small enough to copy directly into your project. The architecture of the validated coding assistant includes three layers Figure 1 . At the top, the developer IDE sends requests to a NeMo Guardrails proxy that fronts the StarCoder2 NIM, which serves completions from your own GPUs. Commits then flow through a CI verification gate to a reviewer and merge. Merged pull requests feed a Prometheus and Grafana metrics loop, whose escape-rate signal loops back to tighten the NeMo Guardrails policy. The components are intentionally small. Each step is independently useful, so a team can adopt the system incrementally instead of treating self-hosted AI assistance as a single large migration. The important design choice is that the model is not the control plane. The model proposes code, but policy enforcement, dependency verification, source traceability, and outcome measurement live outside the model in systems that engineering teams already trust. This approach maintains an understandable deployment. If a suggestion is blocked, you can inspect the NeMo Guardrails policy. If a package is rejected, you can inspect the dependency scan output. If AI-assisted changes regress, you can inspect the same production metrics you use for human-authored changes. Step 1: Deploy StarCoder2 as an NVIDIA NIM NIM ships StarCoder2 as a container with an OpenAI-compatible endpoint, which is what most integrated development environment IDE assistants expect. Pin the container to a specific version from the NGC catalog https://catalog.ngc.nvidia.com/ , rather than using an unversioned tag. export NGC API KEY=<your-ngc-key export STARCODER NIM VERSION=<latest-tag-from-ngc export LOCAL NIM CACHE=~/.cache/nim mkdir -p "$LOCAL NIM CACHE" docker run -d --name starcoder2-nim \ --gpus all \ --shm-size=16GB \ -e NGC API KEY \ -v "$LOCAL NIM CACHE:/opt/nim/.cache" \ -u $ id -u \ -p 8000:8000 \ nvcr.io/nim/bigcode/starcoder2-7b:${STARCODER NIM VERSION} Next, verify the endpoint: curl http://localhost:8000/v1/health/ready curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "bigcode/starcoder2-7b", "prompt": "def fibonacci n: int - int:\n ", "max tokens": 64 }' No source code is leaving your network at this point. The model endpoint is also the same artifact you can pin, scan, and promote through your internal platform catalog. For a pilot, run the endpoint on a single shared GPU host and restrict access to one team. For a broader rollout, put the NIM behind your internal service mesh or load balancer, keep the NGC key in your secrets manager, and publish the pinned image version through the same platform channel you use for other developer services. Step 2: Wire the StarCoder2 NIM into the IDE Most modern IDE assistants accept a custom OpenAI-compatible base URL. For example, Continue can point directly at the local NIM endpoint: { "models": { "title": "StarCoder2 NIM self-hosted ", "provider": "openai", "model": "bigcode/starcoder2-7b", "apiBase": "http://localhost:8000/v1", "apiKey": "not-needed-for-local-nim" } , "tabAutocompleteModel": { "title": "StarCoder2 NIM autocomplete ", "provider": "openai", "model": "bigcode/starcoder2-7b", "apiBase": "http://localhost:8000/v1" } } Cursor, Cline, and other tools that support a custom OpenAI endpoint follow the same pattern. For teams that already have an IDE standard, keep the NIM endpoint stable and make the IDE adapter the replaceable part. That way, the organization can compare assistants without changing the model-serving, policy, CI, or metrics layers underneath. Step 3: Install NVIDIA NeMo Guardrails in front of the NIM This step introduces validation. NeMo Guardrails sits between the IDE and the NIM and can refuse requests that violate a written task policy. For example, “Do not generate authentication, payment, or cryptography code.” This maps directly to the human-only paths many teams already define in AI usage policies. pip install nemoguardrails openai mkdir -p code-rails/config Now, create code-rails/config/config.yml : models: - type: main engine: openai parameters: base url: http://localhost:8000/v1 api key: not-needed-for-local-nim model: bigcode/starcoder2-7b rails: input: flows: - check task policy prompts: - task: self check input content: | Decide whether the following code request touches any of: - authentication / login / session handling - payment processing - cryptography / key material - file paths under src/security/, src/auth/, or src/payments/ Reply with only "YES" or "NO". Request: {{ user input }} Then create code-rails/config/rails.co : php define flow check task policy $allowed = execute self check input if not $allowed bot refuse with policy message stop define bot refuse with policy message "This path is marked human-only by your AI usage policy. Please author it manually and request review." The built-in self check input action renders the self check input prompt, calls the model, and returns a Boolean: False when the prompt answers YES the request touches a human-only path . The flow refuses whenever the request is not allowed. Next, run NeMo Guardrails as the OpenAI-compatible proxy: nemoguardrails server --config=code-rails/config --port=8100 Then point the IDE at http://localhost:8100/v1 instead of http://localhost:8000/v1 . Requests that touch restricted paths are intercepted before they reach the model, and the developer receives a clear policy message instead of a risky completion. Reading the sequence in Figure 2, NeMo Guardrails runs self check input before the model is ever called. A request that touches a human-only path is refused on the spot and the NIM is never reached, while a permitted request is forwarded to the NIM and the completion is returned to the IDE. Start with a conservative policy. Good first candidates for human-only paths include authentication, authorization, payment processing, cryptography, deployment manifests, and incident-response automation. Teams can relax the policy later after they have enough review data to prove that the assistant is safe in a narrower area. Step 4: Add the CI verification gate Generation controls in the IDE are necessary but not sufficient. CI is where you catch package hallucinations, license drift, secret leakage, and insecure patterns before a reviewer becomes responsible for them. Figure 3 reads left to right. A pull request PR labeled ai-assisted passes through unit tests, SAST, a secret scan, the hallucinated-dependency scan, and a license scan, layering model-specific checks on top of the normal test suite. If every check passes, the change goes to a human reviewer; if any step fails, the pull request is blocked and the offending stage is named. Add an AI-assisted PR workflow that runs only when the PR carries an ai-assisted label. Rather than reinventing each check, wire in maintained open-source tools: name: ai-assisted-pr-checks on: pull request: types: opened, synchronize, labeled jobs: verification: if: contains github.event.pull request.labels. .name, 'ai-assisted' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: actions/setup-python@v5 with: python-version: "3.11" - name: Run unit tests run: make test - name: SAST Semgrep run: | pip install semgrep semgrep ci --config p/ci - name: Secret scan uses: gitleaks/gitleaks-action@v2 - name: Hallucinated-dependency slopsquatting scan run: | pip install dep-hallucinator dep-hallucinator scan requirements.txt - name: License scan run: | pip install -r requirements.txt pip install pip-licenses pip-licenses --partial-match --fail-on="GPL;AGPL;LGPL;SSPL" The dependency scan is the highest-leverage step because it targets a failure mode unique to code models, now commonly called slopsquatting . The model invents a plausible-looking package name, an attacker registers that name in a public registry, and the hallucinated dependency ships real malware to anyone who installs the suggestion. Several maintained scanners detect this by checking every newly added dependency against the real registry and flagging names that do not exist, were registered very recently, or closely resemble a popular package: : PyPI, npm, Maven, crates.io, and Go; naming heuristics; SBOM output; CI exit codes dep-hallucinator : Python, npm, and Go; PR-diff aware slopgate slopgate scan . --added-only --base-ref origin/main ; uploads SARIF to Security tab: Combines CVE scanning, slopsquatting detection, and SBOM generation in a single pass XBOM Pin whichever tool you choose to a specific version, exactly as you would pin any other dependency. Note that the StarCoder2 NIM container already ships a signed SBOM and VEX record for the model image itself, so these scanners cover your application’s dependency manifests while NVIDIA covers the model container. For more details, see Securely Deploy AI Models with NVIDIA NIM https://developer.nvidia.com/blog/securely-deploy-ai-models-with-nvidia-nim/ . For license drift, using pip-licenses https://pypi.org/project/pip-licenses/ fails the build when a newly pulled dependency carries a copyleft family your legal team blocks. For a richer, multi-ecosystem bill of materials, you can show differences across references and generate one with Syft https://github.com/anchore/syft or cyclonedx-bom https://pypi.org/project/cyclonedx-bom/ . For air-gapped CI where you cannot add a third-party tool, the same check is about 40 lines of standard library. First diff the manifest between the base and head refs. Then query the registry for each newly added name and fail on a 404 invented , a first-publish date under your threshold likely typo-squat , or a copyleft license. Treat a handrolled version as a fallback, not a replacement for the maintained scanners previously mentioned. For GitLab, the equivalent job can live in .gitlab-ci.yml with a rule that matches CI MERGE REQUEST LABELS against ai-assisted . The same tools run unchanged. Keep this gate stricter than the baseline pipeline. AI-assisted PRs should pass the normal test suite plus checks targeted at model failure modes, including hallucinated packages, secrets copied from prompts, unsafe examples lifted from public code, and dependency licenses that a human reviewer would not catch by eye. Step 5: Make AI assistance traceable You cannot measure what you cannot tag. Install a prepare-commit-msg hook so commits authored with help from the assistant carry a structured trailer: bash /usr/bin/env bash COMMIT MSG FILE=$1 if -n "$AI ASSISTANT" ; then { echo echo "AI-Assistant: ${AI ASSISTANT}" echo "AI-Scope: ${AI SCOPE:-unspecified}" } "$COMMIT MSG FILE" fi Activate this once per repo: git config core.hooksPath .githooks chmod +x .githooks/prepare-commit-msg Then export AI ASSISTANT=starcoder2-nim in the shell from which the IDE is launched. Every assistant-influenced commit now carries a trailer, and CI can autolabel the PR by grepping commit messages. Do not use this trailer as a blame mechanism. Its job is measurement. The useful question is not whether a particular developer used AI, but whether AI-assisted changes have a different review latency, rollback rate, or defect escape rate than the baseline. Step 6: Wire outcome metrics The acceptance rate is not enough. It conflates trivial completions with meaningful engineering work. The metrics that matter are defect escape rate, rollback frequency, review latency, and incident count, broken out by AI-assisted versus baseline. A minimal Prometheus exporter can start with the following two counters: python from prometheus client import Counter, start http server escape = Counter "ai assisted defects escaped total", "Defects shipped to prod from AI-assisted PRs", "severity" rollback = Counter "ai assisted rollbacks total", "Reverts of AI-assisted PRs" Fill out the exporter to poll merged ai-assisted PRs, increment escape from linked incident issues and rollback from revert PRs, and expose /metrics on port 9101 for Prometheus to scrape. Track which PRs and incidents you have already counted so repeated polls do not inflate the counters. Scrape the exporter from your existing Prometheus and graph the AI-assisted series next to baseline. If the AI-assisted escape rate trends above baseline for two consecutive weeks, tighten the task policy, add a CI gate, or pause the rollout. Figure 4 shows this process as a serpentine loop. The top row runs left to right, where merged AI-assisted pull requests and incident signals feed the Prometheus exporter, which Prometheus scrapes and Grafana visualizes. The signal then drops from the Grafana dashboard into the bottom row, which runs right to left and compares AI-assisted against baseline on defect escape rate, rollback frequency, review latency, and incident count, ending by tightening the task policy or pausing the rollout when the escape rate stays high. Optional: Domain-adapt the model with NVIDIA NeMo Framework Off-the-shelf StarCoder2 can hallucinate internal APIs because it has never seen them. NVIDIA ChipNeMo research https://research.nvidia.com/publication/2023-10 chipnemo-domain-adapted-llms-chip-design showed how continued pretraining on a domain-specific corpus, supervised fine-tuning, and retrieval customization can improve assistant quality for specialized engineering domains. If you have a large internal corpus, NeMo Framework https://docs.nvidia.com/nemo-framework/ provides the building blocks for continued pretraining, supervised fine-tuning, and retrieval customization. The domain-adapted model can then be packaged as a NIM and dropped into Step 1 without changing guardrails, CI, traceability, or metrics. This separation makes the architecture durable. You can begin with StarCoder2, later swap in a stronger code-tuned model, and eventually deploy a domain-adapted NIM without rewriting the validation pipeline around it. Step 7: Verify the full loop Before handing the setup to a team, follow the steps below to run one smoke test: - Ask the assistant to write a helper in a permitted path. Confirm a suggestion arrives. - Ask it to modify src/auth/login.py . Confirm that NeMo Guardrails refuses with the policy message. - Open a PR with an AI-assisted change that introduces a fake package name. Confirm the slopsquatting scan fails the check. - Open a clean AI-assisted PR. Confirm the ai-assisted label triggers the full verification job and the commit carries the AI-Assistant trailer. - Revert an AI-assisted PR. Confirm the rollback counter increments. Any failure points at a single component you can fix in isolation. That is the value of building the pipeline as separable parts. Final steps Pin the NIM container version and add it to your platform team’s standard catalog. Move NeMo Guardrails behind a load balancer if more than a handful of developers will use it. Layer your existing static analysis and test gates behind the AI-assisted verification stage so AI-assisted PRs pass a strict superset of baseline checks. If the assistant starts missing internal APIs, evaluate NeMo Framework for domain adaptation and NVIDIA AI Workbench https://www.nvidia.com/en-us/deep-learning-ai/solutions/data-science/workbench/ for reproducible per-developer environments. Learn more A trustworthy code assistant is a pipeline, not a model. Serving StarCoder2 as a NIM keeps your source on your own GPUs. NeMo Guardrails refuses requests for human-only paths before they ever reach the model. The CI gate catches hallucinated packages, leaked secrets, and license drift before a reviewer becomes responsible for them. Commit trailers make AI-assisted changes traceable, and outcome metrics tell you whether those changes are improving or hurting your defect rate. Because policy, verification, traceability, and measurement all live outside the model, you can adopt the layers one at a time and swap in a stronger or domain-adapted model later without rewriting the validation around it. To learn more about the NVIDIA components used in this tutorial, check out these related resources: StarCoder2 NIM https://build.nvidia.com/bigcode/starcoder2-7b/deploy : View the model card, API reference, and container deployment steps for Step 1. NeMo Guardrails https://github.com/NVIDIA/NeMo-Guardrails : See the rails, flows, and actions behind the task policy in Step 3. Securely Deploy AI Models with NVIDIA NIM https://developer.nvidia.com/blog/securely-deploy-ai-models-with-nvidia-nim/ : Read about the signed SBOM and VEX records that complement the dependency scanning in Step 4. NeMo Framework https://docs.nvidia.com/nemo-framework/ : Continued pretraining, supervised fine-tuning, and retrieval customization for domain-adapting the model.