I've been on both sides of the GitHub issue wall.
As a contributor, I've spent time writing what I thought was a decent bug report, only to get a one-liner back: "Can you share a repro?" Three days later I reply. No response. Issue goes stale.
As someone who's poked around in open source repos, I've seen maintainers close issues with a "not enough info" label that they clearly copy-pasted from a template. You can't blame them — doing that manually for every bad issue is exhausting.
That friction is what I tried to fix.
Hackathon Raptors ran the Slop Scan hackathon with a clear brief: build something that detects low-quality content. Most participants went after social media posts or AI-generated text.
I looked at GitHub issues.
They're a form of content too, and they have a very clear quality signal: does this issue give a maintainer enough information to act on it, or not? That's binary and measurable. I could build a scoring system around it.
The result was Signal-OSS — a GitHub Action that scores every new issue from 0 to 10, always posts a structured comment with a checklist tailored to that score, and edits that comment if the contributor updates their issue.
I placed 5th overall and won the Best Detection Accuracy Award. Here's what I actually built.
Most repos handle bad issues in one of two ways:
Neither approach tells the contributor what is actually missing. That's the gap Signal-OSS targets.
The goal isn't to reject bad issues — it's to convert them into good ones by being specific about what's missing.
When a new issue is opened, Signal-OSS runs and always posts a comment — within seconds. What changes based on the score is the content of that comment: a high-scoring issue gets a checklist confirming what's already present, while a low-scoring one gets a checklist of what's specifically missing from that issue.
a newly opened GitHub issue with the bot comment appearing automatically at the bottom
The comment isn't generic. It's generated based on the actual signals detected in that issue — not a boilerplate reply, but a specific breakdown the contributor can act on.
This is the part I find most interesting to explain.
The naive approach would be: send the issue to an LLM, ask it to score it, done. That works, but it couples every issue to an external API call — slow, potentially costly, and requires a key just to get started.
Signal-OSS takes a different approach: heuristics first, LLM only as an optional tiebreaker.
Before any LLM is involved, the action checks 7 rule-based signals:
Each signal contributes to the score. Issues that pass most of these signals will likely land above 7. Issues that fail most will land below 3. Both cases get handled without any LLM call.
Issues that land in the 4–6 range are genuinely ambiguous. Maybe the repro steps are there but vague. Maybe there's a stack trace but no environment context. For these, the heuristics can't reliably decide.
This is the only zone where an LLM can get called — and only if you've provided an API key. Signal-OSS is BYOK: bring your own LLM key if you want the smarter scoring, leave it out and the action still works. For the ambiguous zone specifically, omitting the key falls back to the standard heuristic checklist, which is good enough for most repos.
The result: the action is fully functional with zero external API keys. The LLM layer is an opt-in upgrade, not a requirement.
the bot's comment showing the checklist andneeds-info
label option
the bot's comment showing thesignal-oss-ignore
label option
The bot also:
needs-info
label automatically on low scoressignal-oss-ignore
label option — if a maintainer adds this to any issue, the bot stays completely silent for that issueThis was a deliberate design constraint I set early: the activation path has to be a single YAML file. No secrets to configure, no dashboard to sign up for.
name: Signal-OSS Issue Triage
on:
issues:
types: [opened, reopened]
permissions:
contents: read
issues: write
jobs:
triage:
runs-on: ubuntu-latest
if: github.actor != 'github-actions[bot]'
steps:
- name: Triage issue
uses: Divyesh-5981/signal-oss@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
model: 'none'
The GITHUB_TOKEN
is automatically provided by GitHub Actions — no setup needed. The action works fully without the LLM key, using the standard heuristic checklist for all issues including the ambiguous 4–6 zone. If you add a key, ambiguous issues get LLM-refined scoring instead. Drop the file into .github/workflows/
and you're done either way.
the summary view in the Actions tab showing the run result and the score output
I benchmarked Signal-OSS on real issues from three large repositories: microsoft/vscode
, facebook/react
, and rust-lang/rust
.
| Metric | Score |
|---|---|
| Precision | 0.66 |
| Recall | 0.93 |
| F1 | 0.77 |
The precision being lower than recall was intentional. Here's the tradeoff:
High recall means the system rarely misses a bad issue. If something is genuinely low-quality, Signal-OSS will catch it.
Lower precision means it sometimes flags issues that are actually fine. This results in false positives — good issues that get a comment asking for more detail.
For this use case, that's the right tradeoff. Missing a bad issue (false negative) wastes maintainer time. Flagging a good issue (false positive) is slightly annoying for the contributor but recoverable — they can see the score is borderline and move on, or the maintainer can apply signal-oss-ignore
.
If I'd optimized for precision, I'd be missing bad issues constantly. That defeats the purpose.
A few honest notes from the hackathon:
The 4–6 threshold was arbitrary. I chose it based on intuition and tested it during the benchmark, but a configurable threshold would make Signal-OSS more useful across different repo cultures. A high-volume OSS project might want stricter rules than a small internal tool.
The heuristics are English-biased. All 7 signal checks were designed for English-language issues. Issues written in other languages will score lower by default, which isn't fair. This is worth fixing before any serious adoption.
I benchmarked on large repos. The issue quality distribution in vscode
or react
is probably different from smaller, less-structured projects. The benchmark numbers may not generalize.
The repo is linked below. If you maintain an open-source project and issue triage is eating your time, this is worth a look.
Two things I'd find genuinely useful from you:
👇 Drop your thoughts below. And if this was useful, the ❤️ helps other maintainers find it.
Built during the Slop Scan hackathon organized by Hackathon Raptors. Repo in the comments.