I Built a GitHub Bot That Reviews My Pull Requests Using a Local LLM

Creation of a GitHub bot that uses a local large language model (LLM) via Ollama to automatically review pull requests. When a PR is opened, the bot receives a webhook, retrieves the code diff using GitHub's REST API, sends it to the locally running LLM for analysis, and posts the review as a comment. The author explains the technical setup, including configuring a GitHub App, handling authentication with short-lived installation tokens, and splitting large diffs into smaller chunks to fit within the LLM's context window.

Last month I pushed a bug to production which I would have caught if I had somebody look at my code. Well, it's not like I was the only one working on the project, so I had to read it myself and then forget about it, only to spend an annoying Tuesday trying to figure out why it didn't ship. I was thinking about Ollama. I had heard about it a couple of times but never really needed to use it for anything. So this weekend I built a GitHub bot that reviews pull requests with a local LLM, without sending any code out of the box. It makes a comment on the PR with what it finds. Here's how I built it. What the bot actually does When you open a pull request, GitHub fires a webhook. The bot receives it, pulls the diff for each changed file using GitHub's REST API, sends that diff to a locally running LLM via Ollama, and posts the review back as a comment on the PR. The output looks like this on your PR: Project Structure: self-hosted-ai-code-review-bot-for-github-prs/ ├── src/ │ ├── server.js │ ├── github.js │ ├── ollama.js │ ├── diffParser.js │ └── chunker.js ├── private-key.pem ├── .env ├── package-lock.json └── package.json Setting up the GitHub App first Before any code, you need a GitHub App. A GitHub App has its own identity, which gets installed on specific repos, and uses short-lived installation tokens instead of long-lived credentials. Go to GitHub → Settings → Developer Settings → GitHub Apps → New GitHub App. Permissions you need: - Pull requests: Read & write - Issues: Read & write - Metadata: Read-only Under webhook events, subscribe to Pull request. Set the webhook URL to your ngrok address + /webhook we'll come back to that . Generate and download the private key .pem file . GitHub uses this to sign the installation tokens. Clone the repo and install dependencies: git clone <your-repo-url cd <repo-name npm install Your .env : PORT=3000 GITHUB APP ID=your app id GITHUB PRIVATE KEY PATH=./privatekey.pem WEBHOOK SECRET=your webhook secret OLLAMA MODEL=deepseek-coder or llama3, whichever you're running Then install the app on your repository via the Install App option. github.js — auth and API calls GitHub Apps don't use static tokens. You sign a JWT with your private key, exchange it for a short-lived installation token, and use that to make API calls. @octokit/auth-app handles all of this: The auth layer GitHub Apps use installation tokens, not static keys. You sign a JWT with your private key, exchange it for a short-lived token, use that token for API calls. @octokit/auth-app handles the whole flow: js export async function getOctokit installationId { const privateKey = fs.readFileSync process.env.GITHUB PRIVATE KEY PATH, "utf8" ; const auth = createAppAuth { appId: process.env.GITHUB APP ID, privateKey, installationId } ; const installationAuth = await auth { type: "installation" } ; return new Octokit { auth: installationAuth.token } ; } You pass the installationId from the webhook payload. GitHub includes it in every event so you always know which installation triggered it. The other two functions in github.js are straightforward: one calls octokit.pulls.listFiles to get the changed files with their diffs, the other calls octokit.issues.createComment to post the final review. Filtering and chunking the diff Not every file in a PR deserves review time. package-lock.json , .vscode/ settings, minified files, we skip them. GitHub also sometimes returns files without a patch field binary files, files too large to diff , those get skipped too. For files that do make it through, large diffs need to be split before sending to the LLM. Context windows are limited and model quality drops toward the end of long inputs: js export function chunkDiff diff, maxSize = 1500 { const chunks = ; if diff return chunks; let current = ""; const lines = diff.split "\n" ; for const line of lines { if current + line .length maxSize { chunks.push current ; current = ""; } current += line + "\n"; } if current chunks.push current ; return chunks; } 1500 characters per chunk works reliably with both Llama 3 and DeepSeek Coder. Each chunk gets its own Ollama call, results get concatenated per file. The prompt is important This is where I spent the most time, and where the whole thing either works or doesn't. My first version was something unclear like "review this diff and point out issues." The output was not good. The model would write paragraphs explaining what the file does, invent security vulnerabilities that weren't in the code, reference issue numbers that don't exist, and end with generic advice like "make sure to add proper error handling throughout your application." The problem is that without constraints, the model tries to be helpful in every direction at once. It doesn't know you only care about what changed. So you have to be as explicit about what you don't want as what you do: js export async function reviewWithOllama diffChunk { const prompt = You are an AI GitHub pull request reviewer. You are reviewing ONLY the provided git diff. STRICT RULES: - Review ONLY changed lines from the diff. - Ignore unchanged code completely. - Do NOT explain the whole application. - Do NOT give generic software engineering advice. - Do NOT hallucinate missing features. - Do NOT invent security issues. - Do NOT invent issue IDs, ticket numbers, PR references, or metadata. - Do NOT mention issues outside the diff. - Every issue MUST directly relate to a changed line. - Keep response concise. - If no real issue exists, reply exactly: "No significant issues found." Focus ONLY on: 1. Potential bugs introduced 2. Style or readability issues introduced 3. Performance issues introduced 4. Security risks directly introduced 5. Small improvement suggestions directly related to the diff Diff: ${diffChunk} ; const response = await fetch "http://localhost:11434/api/generate", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify { model: process.env.OLLAMA MODEL, prompt, stream: false, options: { temperature: 0 } } } ; const data = await response.json ; return data.response; } Once I added the negative rules, the output became useful. The model stopped inventing context and started pointing at specific changed lines. temperature: 0 — code review isn't creative work. You want the model to be deterministic, same diff producing consistent output. It also reduces hallucinations noticeably. The webhook handler Express receives the event, checks if it's worth handling, and orchestrates everything: js app.post "/webhook", async req, res = { try { const event = req.headers "x-github-event" ; const action = req.body.action; if event == "pull request" return res.status 200 .send "Ignored" ; if "opened", "synchronize", "reopened" .includes action { return res.status 200 .send "Ignored action" ; } const installationId = req.body.installation.id; const owner = req.body.repository.owner.login; const repo = req.body.repository.name; const pullNumber = req.body.pull request.number; const octokit = await getOctokit installationId ; const files = await getPullRequestFiles octokit, owner, repo, pullNumber ; let finalReview = 🤖 AI Code Review\n\n ; for const file of files { if shouldReviewFile file continue; const chunks = chunkDiff file.patch ; let fileReview = ""; for const chunk of chunks { fileReview += await reviewWithOllama chunk + "\n\n"; } finalReview += File: ${file.filename}\n\n${fileReview}---\n\n ; } await postReviewComment octokit, owner, repo, pullNumber, finalReview ; res.status 200 .send "OK" ; } catch error { console.error "Webhook error:", error ; res.status 500 .send "Error processing webhook" ; } } ; Running it Install Ollama from ollama.com https://ollama.com and pull a model: ollama run llama3 or, better for code ollama run deepseek-coder Start the server and expose it with ngrok: npm run dev ngrok http 3000 Copy the ngrok HTTPS URL, update your GitHub App's webhook URL to https://your-ngrok-url/webhook https://your-ngrok-url/webhook , push a branch, open a PR. CPU vs GPU: Depending on hardware, local inference on CPU may take several seconds per diff chunk. Llama 3 8B on a laptop CPU takes 10–20 seconds per diff chunk. For a background bot that's fine, you open the PR, switch tabs, comment shows up a minute later. With an NVIDIA GPU, Ollama picks it up automatically and it's under 2 seconds. Deploying it properly ngrok dies when you close your terminal. For anything persistent, a cheap VPS works. DigitalOcean's $6/month droplet, small EC2 instance, whatever. Install Node.js and Ollama on it, put Nginx in front with SSL, point your webhook URL at the domain. Tools like GitHub Copilot Code Review send your diff to external servers. With this setup the diff goes from GitHub's API to your own machine into your local model which means nothing leaves your infrastructure. For proprietary codebases that's not a minor thing. Where the idea came from I came across VickyBytes https://vickybytes.com a few weeks back. It's a platform that posts structured project ideas in Creator Labs, where you build something, write about it, and earn from it if the content is good enough. This was one of those. Having a concrete spec helped. "Play with Ollama" had been on my list for months and never happened. A specific project with a clear outcome actually got it done. The bot runs on my own repos now, so that worked out. If you write technical content and want structured project ideas to build around, visit vickybytes.com https://vickybytes.com . Full source is on GitHub https://github.com/Shresthap21/Self-Hosted-AI-Code-Review-Bot-for-GitHub-PRs . The prompt is where most of the interesting experimentation happens. That's where you'll actually learn how local LLMs behave in practice.