{"slug": "create-a-custom-ai-pr-reviewer-from-scratch-with-github-actions", "title": "Create a Custom AI PR Reviewer from scratch with GitHub Actions", "summary": "A developer built a custom AI code reviewer using GitHub Actions and the Gemini API. The lightweight, dependency-free automation triggers on pull requests, extracts the git diff, sends it to Gemini for structured review, and posts inline comments. The project demonstrates that a fully customized AI PR reviewer can be created without third-party platforms.", "body_md": "Every time I submit a Pull Request (PR) to an open-source project, I watch the CI pipeline kick off. More and more often, an AI assistant like CodeRabbit, Copilot, or Claude pops up a few minutes later to review my changes, categorize bugs, or suggest updates.\n\nIt got me wondering: **How hard is it to build one of these from scratch?**\n\nAs it turns out, you don't need a heavy, expensive third-party platform. You can build a fully customized, line-by-line AI code reviewer directly inside GitHub Actions using a short Python script and the Gemini API.\n\nHere is exactly how it works, what the code looks like, and a breakdown of how each piece functions.\n\nThe goal is to create a lightweight, dependency-free automation loop that acts like a senior engineer on your team.\n\n**Trigger on PR Events:** Step 1.\n\nThe workflow triggers whenever a PR is opened, updated, or reopened.\n\n**Extract the Git Diff:** Step 2.\n\nWe isolate the exact line changes and trim them to fit comfortably within LLM context constraints.\n\n**Send to Gemini with Structured Output:** Step 3.\n\nAn inline Python script passes the diff to Gemini, using a strict JSON schema to guarantee we get back structured comments tied to specific files and lines.\n\n**Post Inline Comments:** Step 4.\n\nA GitHub script reads the JSON payload and attaches native code-review comments directly onto the modified lines in the PR.\n\nCreate a file named `.github/workflows/ai-review.yml`\n\nin your repository and paste the following configuration:\n\n```\nname: AI PR Review\n\non:\n  pull_request:\n    types: [opened, synchronize, reopened]\n\npermissions:\n  contents: read\n  pull-requests: write\n\njobs:\n  ai-review:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n        with:\n          fetch-depth: 0\n\n      - name: Get PR diff\n        env:\n          BASE_SHA: ${{ github.event.pull_request.base.sha }}\n          HEAD_SHA: ${{ github.event.pull_request.head.sha }}\n        run: |\n          git diff \"$BASE_SHA...$HEAD_SHA\" > diff.txt\n          head -c 800000 diff.txt > diff_trimmed.txt\n\n      - name: Ask Gemini for a review\n        env:\n          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}\n        run: |\n          python3 - <<'PYEOF'\n          import json, os, re, sys, time\n          import urllib.error\n          import urllib.request\n\n          MAX_DIFF_BYTES = 800_000\n\n          def write_outputs(status, summary, comments):\n              open(\"status.txt\", \"w\").write(status)\n              open(\"review.md\", \"w\").write(summary)\n              open(\"comments.json\", \"w\").write(json.dumps(comments))\n\n          original_bytes = os.path.getsize(\"diff.txt\")\n          truncated = original_bytes > MAX_DIFF_BYTES\n\n          with open(\"diff_trimmed.txt\", errors=\"replace\") as f:\n              diff = f.read()\n\n          if not diff.strip():\n              write_outputs(\"clean\", \"No changes to review.\", [])\n              sys.exit(0)\n\n          def parse_commentable_lines(diff_text):\n              valid = {}\n              current_file, new_line = None, None\n              for line in diff_text.splitlines():\n                  if line.startswith(\"+++ \"):\n                      raw = line[4:].strip()\n                      current_file = None if raw == \"/dev/null\" else raw[2:]  # strip \"b/\"\n                      if current_file:\n                          valid.setdefault(current_file, set())\n                      new_line = None\n                      continue\n                  m = re.match(r\"^@@ -\\d+(?:,\\d+)? \\+(\\d+)(?:,\\d+)? @@\", line)\n                  if m:\n                      new_line = int(m.group(1))\n                      continue\n                  if current_file is None or new_line is None:\n                      continue\n                  if line.startswith(\"+\"):\n                      valid[current_file].add(new_line)  # only new/added lines are offered\n                      new_line += 1\n                  elif line.startswith(\" \"):\n                      new_line += 1\n              return valid\n\n          commentable = parse_commentable_lines(diff)\n\n          prompt_text = \"\"\"You are a senior software engineer reviewing a pull request diff.\n\n          Focus strictly on:\n          1. Architecture & Logic: bugs, security issues, missing error handling, race conditions, lifecycle conflicts.\n          2. UI & Responsiveness\n          3. UX & Information Density.\n          4. Maintainability: structural and style problems.\n\n          Only flag lines that are actually part of the diff (added or modified lines). Reference the NEW file's line numbers.\n          Zero emojis. Be brief, direct, pragmatic.\n          If there is nothing significant, return status \"clean\" with an empty comments array.\n          \"\"\" + diff\n\n          response_schema = {\n              \"type\": \"OBJECT\",\n              \"properties\": {\n                  \"status\": {\"type\": \"STRING\", \"enum\": [\"clean\", \"issues_found\"]},\n                  \"summary\": {\"type\": \"STRING\"},\n                  \"comments\": {\n                      \"type\": \"ARRAY\",\n                      \"items\": {\n                          \"type\": \"OBJECT\",\n                          \"properties\": {\n                              \"file\": {\"type\": \"STRING\"},\n                              \"line\": {\"type\": \"INTEGER\"},\n                              \"severity\": {\"type\": \"STRING\", \"enum\": [\"high\", \"medium\", \"low\"]},\n                              \"comment\": {\"type\": \"STRING\"},\n                          },\n                          \"required\": [\"file\", \"line\", \"comment\"],\n                      },\n                  },\n              },\n              \"required\": [\"status\", \"summary\", \"comments\"],\n          }\n\n          req = urllib.request.Request(\n              \"https://generativelanguage.googleapis.com/v1beta/models/gemini-3.5-flash:generateContent\",\n              data=json.dumps({\n                  \"contents\": [{\"parts\": [{\"text\": prompt_text}]}],\n                  \"generationConfig\": {\n                      \"responseMimeType\": \"application/json\",\n                      \"responseSchema\": response_schema,\n                  },\n              }).encode(),\n              headers={\n                  \"Content-Type\": \"application/json\",\n                  \"x-goog-api-key\": os.environ[\"GEMINI_API_KEY\"],\n              },\n          )\n\n          def call_with_retries(request, max_attempts=3, base_delay=2):\n              last_err = None\n              for attempt in range(1, max_attempts + 1):\n                  try:\n                      with urllib.request.urlopen(request, timeout=180) as resp:\n                          return json.loads(resp.read())\n                  except urllib.error.HTTPError as e:\n                      last_err = e\n                      if e.code in (429, 500, 502, 503, 504) and attempt < max_attempts:\n                          time.sleep(base_delay * attempt)\n                          continue\n                      raise\n                  except urllib.error.URLError as e:\n                      last_err = e\n                      if attempt < max_attempts:\n                          time.sleep(base_delay * attempt)\n                          continue\n                      raise\n              raise last_err\n\n          try:\n              payload = call_with_retries(req)\n\n              candidates = payload.get(\"candidates\") or []\n              if not candidates:\n                  reason = payload.get(\"promptFeedback\", {}).get(\"blockReason\", \"unknown\")\n                  raise ValueError(f\"no candidates returned (blockReason={reason})\")\n              if \"content\" not in candidates[0] or \"parts\" not in candidates[0][\"content\"]:\n                  raise ValueError(\"response missing content/parts\")\n\n              raw_text = candidates[0][\"content\"][\"parts\"][0].get(\"text\", \"\").strip()\n              raw_text = re.sub(r\"^```\n{% endraw %}\njson\\s*|\\s*\n{% raw %}\n```$\", \"\", raw_text)  # in case it wraps anyway\n              parsed = json.loads(raw_text)\n\n              status = parsed.get(\"status\", \"issues_found\")\n              summary = parsed.get(\"summary\", \"\").strip() or \"Review complete.\"\n              raw_comments = parsed.get(\"comments\", [])\n\n              safe_comments = []\n              for c in raw_comments:\n                  f, ln = c.get(\"file\"), c.get(\"line\")\n                  if f in commentable and ln in commentable[f]:\n                      severity = c.get(\"severity\", \"medium\").upper()\n                      safe_comments.append({\n                          \"path\": f,\n                          \"line\": ln,\n                          \"side\": \"RIGHT\",\n                          \"body\": f\"**[{severity}]** {c.get('comment', '').strip()}\",\n                      })\n\n              if truncated:\n                  summary = (\n                      f\"Note: diff exceeded {MAX_DIFF_BYTES // 1000}KB and was truncated — \"\n                      f\"this review may not cover the full PR.\\n\\n\" + summary\n                  )\n\n              write_outputs(status, summary, safe_comments)\n\n          except urllib.error.HTTPError as e:\n              detail = e.read().decode(errors=\"replace\")[:500]\n              print(f\"::warning::Gemini API HTTP {e.code}: {detail}\", file=sys.stderr)\n              write_outputs(\"skipped\", f\"AI review skipped: Gemini API returned HTTP {e.code}.\", [])\n\n          except urllib.error.URLError as e:\n              print(f\"::warning::Gemini API network error: {e}\", file=sys.stderr)\n              write_outputs(\"skipped\", f\"AI review skipped: network error contacting Gemini API ({e.reason}).\", [])\n\n          except (KeyError, IndexError, ValueError, json.JSONDecodeError) as e:\n              print(f\"::warning::Gemini API unexpected response: {e}\", file=sys.stderr)\n              write_outputs(\"skipped\", f\"AI review skipped: unexpected response from Gemini API ({e}).\", [])\n          PYEOF\n\n      - name: Submit PR Review\n        uses: actions/github-script@v7\n        with:\n          script: |\n            const fs = require('fs');\n\n            const status = fs.readFileSync('status.txt', 'utf8').trim();\n            const summary = fs.readFileSync('review.md', 'utf8');\n            let comments = [];\n            try {\n              comments = JSON.parse(fs.readFileSync('comments.json', 'utf8'));\n            } catch (e) {\n              comments = [];\n            }\n\n            const reviewEvent = (status === 'issues_found' && comments.length > 0)\n              ? 'REQUEST_CHANGES'\n              : 'COMMENT';\n\n            const base = {\n              owner: context.repo.owner,\n              repo: context.repo.repo,\n              pull_number: context.issue.number,\n              event: reviewEvent,\n            };\n\n            try {\n              await github.rest.pulls.createReview({ ...base, body: summary, comments });\n            } catch (e) {\n              core.warning(`Inline comments rejected (${e.message}); falling back to summary only.`);\n              await github.rest.pulls.createReview({\n                ...base,\n                body: summary + '\\n\\n_Some inline comments could not be posted._',\n              });\n            }\n```\n\nLet’s look at exactly how each component functions to pull this off seamlessly.\n\nAt the top of the workflow, we define the event triggers and permissions:\n\n`pull_request`\n\ntypes ensure this runs when code changes or a new PR goes live.`pull-requests: write`\n\npermissions. Without this, the runner won't be allowed to leave comments on the PR thread.`fetch-depth: 0`\n\nensures the full git history is pulled down so we can accurately compare branches.\n\n```\ngit diff \"$BASE_SHA...$HEAD_SHA\" > diff.txt\nhead -c 800000 diff.txt > diff_trimmed.txt\n```\n\nInstead of relying on heavy ecosystem dependencies, we pull standard git diff text. Huge pull requests can quickly exceed large language model limits or API limits, so `head -c 800000`\n\nsafely caps the payload size at around 800KB to prevent structural drops.\n\nThe core logic runs in a dependency-free Python step using standard libraries (`urllib`\n\n). It handles three main problems:\n\n`parse_commentable_lines`\n\n):`@@ -line,v +line,v @@`\n\n) to build a map of lines that actually exist in the `prompt_text`\n\nprovides structural guardrails. You can adapt this to fit your specific stack's technical requirements (e.g., advising on UI lifecycles, CSS frameworks, or structural preferences).`response_schema`\n\ndirectly inside the API request options. This forces the `AI model`\n\nto return raw data exactly matching an object containing a status string, markdown summary, and an array of inline comments.The script ends by verifying that the AI's feedback strictly lines up with changed lines and writes the output payload into local text files.\n\n``` js\nconst reviewEvent = (status === 'issues_found' && comments.length > 0) ? 'REQUEST_CHANGES' : 'COMMENT';\n...\nawait github.rest.pulls.createReview({ ...base, body: summary, comments });\n```\n\nThe final step uses the official `actions/github-script`\n\naction to process those text files. If the AI flags real logic errors and populates valid comments, the script marks the PR review state as `REQUEST_CHANGES`\n\n. If everything looks clear, it defaults to a standard tracking `COMMENT`\n\n.\n\nIf a specific line index mismatch occurs, a try/catch block acts as a fallback to post the core high-level summary review anyway so you don't lose the AI insight.\n\nThe beauty of this setup is how easy it is to alter. By adjusting the `prompt_text`\n\nsection in the Python block, you can easily instruct the LLM to write in a specific language, focus tightly on security vulnerabilities, or ignore everything except your performance benchmarks. Drop the `GEMINI_API_KEY`\n\ninto your GitHub repository secrets, and you have an automated, self-hosted code guardrail running completely under your control!", "url": "https://wpnews.pro/news/create-a-custom-ai-pr-reviewer-from-scratch-with-github-actions", "canonical_source": "https://dev.to/thierryrakt/create-a-custom-ai-pr-reviewer-from-scratch-with-github-actions-46p5", "published_at": "2026-07-16 10:37:56+00:00", "updated_at": "2026-07-16 11:03:22.947517+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "ai-tools", "large-language-models", "mlops"], "entities": ["Gemini", "GitHub Actions", "CodeRabbit", "Copilot", "Claude"], "alternates": {"html": "https://wpnews.pro/news/create-a-custom-ai-pr-reviewer-from-scratch-with-github-actions", "markdown": "https://wpnews.pro/news/create-a-custom-ai-pr-reviewer-from-scratch-with-github-actions.md", "text": "https://wpnews.pro/news/create-a-custom-ai-pr-reviewer-from-scratch-with-github-actions.txt", "jsonld": "https://wpnews.pro/news/create-a-custom-ai-pr-reviewer-from-scratch-with-github-actions.jsonld"}}