Create a Custom AI PR Reviewer from scratch with GitHub Actions

A developer built a custom AI code reviewer using GitHub Actions and the Gemini API. The lightweight, dependency-free automation triggers on pull requests, extracts the git diff, sends it to Gemini for structured review, and posts inline comments. The project demonstrates that a fully customized AI PR reviewer can be created without third-party platforms.

Every time I submit a Pull Request PR to an open-source project, I watch the CI pipeline kick off. More and more often, an AI assistant like CodeRabbit, Copilot, or Claude pops up a few minutes later to review my changes, categorize bugs, or suggest updates. It got me wondering: How hard is it to build one of these from scratch? As it turns out, you don't need a heavy, expensive third-party platform. You can build a fully customized, line-by-line AI code reviewer directly inside GitHub Actions using a short Python script and the Gemini API. Here is exactly how it works, what the code looks like, and a breakdown of how each piece functions. The goal is to create a lightweight, dependency-free automation loop that acts like a senior engineer on your team. Trigger on PR Events: Step 1. The workflow triggers whenever a PR is opened, updated, or reopened. Extract the Git Diff: Step 2. We isolate the exact line changes and trim them to fit comfortably within LLM context constraints. Send to Gemini with Structured Output: Step 3. An inline Python script passes the diff to Gemini, using a strict JSON schema to guarantee we get back structured comments tied to specific files and lines. Post Inline Comments: Step 4. A GitHub script reads the JSON payload and attaches native code-review comments directly onto the modified lines in the PR. Create a file named .github/workflows/ai-review.yml in your repository and paste the following configuration: name: AI PR Review on: pull request: types: opened, synchronize, reopened permissions: contents: read pull-requests: write jobs: ai-review: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - name: Get PR diff env: BASE SHA: ${{ github.event.pull request.base.sha }} HEAD SHA: ${{ github.event.pull request.head.sha }} run: | git diff "$BASE SHA...$HEAD SHA" diff.txt head -c 800000 diff.txt diff trimmed.txt - name: Ask Gemini for a review env: GEMINI API KEY: ${{ secrets.GEMINI API KEY }} run: | python3 - <<'PYEOF' import json, os, re, sys, time import urllib.error import urllib.request MAX DIFF BYTES = 800 000 def write outputs status, summary, comments : open "status.txt", "w" .write status open "review.md", "w" .write summary open "comments.json", "w" .write json.dumps comments original bytes = os.path.getsize "diff.txt" truncated = original bytes MAX DIFF BYTES with open "diff trimmed.txt", errors="replace" as f: diff = f.read if not diff.strip : write outputs "clean", "No changes to review.", sys.exit 0 def parse commentable lines diff text : valid = {} current file, new line = None, None for line in diff text.splitlines : if line.startswith "+++ " : raw = line 4: .strip current file = None if raw == "/dev/null" else raw 2: strip "b/" if current file: valid.setdefault current file, set new line = None continue m = re.match r"^@@ -\d+ ?:,\d+ ? \+ \d+ ?:,\d+ ? @@", line if m: new line = int m.group 1 continue if current file is None or new line is None: continue if line.startswith "+" : valid current file .add new line only new/added lines are offered new line += 1 elif line.startswith " " : new line += 1 return valid commentable = parse commentable lines diff prompt text = """You are a senior software engineer reviewing a pull request diff. Focus strictly on: 1. Architecture & Logic: bugs, security issues, missing error handling, race conditions, lifecycle conflicts. 2. UI & Responsiveness 3. UX & Information Density. 4. Maintainability: structural and style problems. Only flag lines that are actually part of the diff added or modified lines . Reference the NEW file's line numbers. Zero emojis. Be brief, direct, pragmatic. If there is nothing significant, return status "clean" with an empty comments array. """ + diff response schema = { "type": "OBJECT", "properties": { "status": {"type": "STRING", "enum": "clean", "issues found" }, "summary": {"type": "STRING"}, "comments": { "type": "ARRAY", "items": { "type": "OBJECT", "properties": { "file": {"type": "STRING"}, "line": {"type": "INTEGER"}, "severity": {"type": "STRING", "enum": "high", "medium", "low" }, "comment": {"type": "STRING"}, }, "required": "file", "line", "comment" , }, }, }, "required": "status", "summary", "comments" , } req = urllib.request.Request "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent", data=json.dumps { "contents": {"parts": {"text": prompt text} } , "generationConfig": { "responseMimeType": "application/json", "responseSchema": response schema, }, } .encode , headers={ "Content-Type": "application/json", "x-goog-api-key": os.environ "GEMINI API KEY" , }, def call with retries request, max attempts=3, base delay=2 : last err = None for attempt in range 1, max attempts + 1 : try: with urllib.request.urlopen request, timeout=180 as resp: return json.loads resp.read except urllib.error.HTTPError as e: last err = e if e.code in 429, 500, 502, 503, 504 and attempt < max attempts: time.sleep base delay attempt continue raise except urllib.error.URLError as e: last err = e if attempt < max attempts: time.sleep base delay attempt continue raise raise last err try: payload = call with retries req candidates = payload.get "candidates" or if not candidates: reason = payload.get "promptFeedback", {} .get "blockReason", "unknown" raise ValueError f"no candidates returned blockReason={reason} " if "content" not in candidates 0 or "parts" not in candidates 0 "content" : raise ValueError "response missing content/parts" raw text = candidates 0 "content" "parts" 0 .get "text", "" .strip raw text = re.sub r"^ {% endraw %} json\s |\s {% raw %} $", "", raw text in case it wraps anyway parsed = json.loads raw text status = parsed.get "status", "issues found" summary = parsed.get "summary", "" .strip or "Review complete." raw comments = parsed.get "comments", safe comments = for c in raw comments: f, ln = c.get "file" , c.get "line" if f in commentable and ln in commentable f : severity = c.get "severity", "medium" .upper safe comments.append { "path": f, "line": ln, "side": "RIGHT", "body": f" {severity} {c.get 'comment', '' .strip }", } if truncated: summary = f"Note: diff exceeded {MAX DIFF BYTES // 1000}KB and was truncated — " f"this review may not cover the full PR.\n\n" + summary write outputs status, summary, safe comments except urllib.error.HTTPError as e: detail = e.read .decode errors="replace" :500 print f"::warning::Gemini API HTTP {e.code}: {detail}", file=sys.stderr write outputs "skipped", f"AI review skipped: Gemini API returned HTTP {e.code}.", except urllib.error.URLError as e: print f"::warning::Gemini API network error: {e}", file=sys.stderr write outputs "skipped", f"AI review skipped: network error contacting Gemini API {e.reason} .", except KeyError, IndexError, ValueError, json.JSONDecodeError as e: print f"::warning::Gemini API unexpected response: {e}", file=sys.stderr write outputs "skipped", f"AI review skipped: unexpected response from Gemini API {e} .", PYEOF - name: Submit PR Review uses: actions/github-script@v7 with: script: | const fs = require 'fs' ; const status = fs.readFileSync 'status.txt', 'utf8' .trim ; const summary = fs.readFileSync 'review.md', 'utf8' ; let comments = ; try { comments = JSON.parse fs.readFileSync 'comments.json', 'utf8' ; } catch e { comments = ; } const reviewEvent = status === 'issues found' && comments.length 0 ? 'REQUEST CHANGES' : 'COMMENT'; const base = { owner: context.repo.owner, repo: context.repo.repo, pull number: context.issue.number, event: reviewEvent, }; try { await github.rest.pulls.createReview { ...base, body: summary, comments } ; } catch e { core.warning Inline comments rejected ${e.message} ; falling back to summary only. ; await github.rest.pulls.createReview { ...base, body: summary + '\n\n Some inline comments could not be posted. ', } ; } Let’s look at exactly how each component functions to pull this off seamlessly. At the top of the workflow, we define the event triggers and permissions: pull request types ensure this runs when code changes or a new PR goes live. pull-requests: write permissions. Without this, the runner won't be allowed to leave comments on the PR thread. fetch-depth: 0 ensures the full git history is pulled down so we can accurately compare branches. git diff "$BASE SHA...$HEAD SHA" diff.txt head -c 800000 diff.txt diff trimmed.txt Instead of relying on heavy ecosystem dependencies, we pull standard git diff text. Huge pull requests can quickly exceed large language model limits or API limits, so head -c 800000 safely caps the payload size at around 800KB to prevent structural drops. The core logic runs in a dependency-free Python step using standard libraries urllib . It handles three main problems: parse commentable lines : @@ -line,v +line,v @@ to build a map of lines that actually exist in the prompt text provides structural guardrails. You can adapt this to fit your specific stack's technical requirements e.g., advising on UI lifecycles, CSS frameworks, or structural preferences . response schema directly inside the API request options. This forces the AI model to return raw data exactly matching an object containing a status string, markdown summary, and an array of inline comments.The script ends by verifying that the AI's feedback strictly lines up with changed lines and writes the output payload into local text files. js const reviewEvent = status === 'issues found' && comments.length 0 ? 'REQUEST CHANGES' : 'COMMENT'; ... await github.rest.pulls.createReview { ...base, body: summary, comments } ; The final step uses the official actions/github-script action to process those text files. If the AI flags real logic errors and populates valid comments, the script marks the PR review state as REQUEST CHANGES . If everything looks clear, it defaults to a standard tracking COMMENT . If a specific line index mismatch occurs, a try/catch block acts as a fallback to post the core high-level summary review anyway so you don't lose the AI insight. The beauty of this setup is how easy it is to alter. By adjusting the prompt text section in the Python block, you can easily instruct the LLM to write in a specific language, focus tightly on security vulnerabilities, or ignore everything except your performance benchmarks. Drop the GEMINI API KEY into your GitHub repository secrets, and you have an automated, self-hosted code guardrail running completely under your control