{"slug": "oss-maintainers-lose-hours-to-vague-issues-i-built-a-github-action-to-fix-that", "title": "OSS Maintainers Lose Hours to Vague Issues. I Built a GitHub Action to Fix That.", "summary": "A developer built Signal-OSS, a GitHub Action that scores new issues from 0 to 10 using heuristics and an optional LLM tiebreaker, automatically posting tailored checklists to help contributors improve low-quality reports. The tool, which won the Best Detection Accuracy Award at the Slop Scan hackathon, is designed to reduce maintainer burnout by converting vague issues into actionable ones without requiring external API keys.", "body_md": "I've been on both sides of the GitHub issue wall.\n\nAs a contributor, I've spent time writing what I thought was a decent bug report, only to get a one-liner back: *\"Can you share a repro?\"* Three days later I reply. No response. Issue goes stale.\n\nAs someone who's poked around in open source repos, I've seen maintainers close issues with a \"not enough info\" label that they clearly copy-pasted from a template. You can't blame them — doing that manually for every bad issue is exhausting.\n\nThat friction is what I tried to fix.\n\n[Hackathon Raptors](https://www.linkedin.com/company/hackathonraptors/) ran the Slop Scan hackathon with a clear brief: build something that detects low-quality content. Most participants went after social media posts or AI-generated text.\n\nI looked at GitHub issues.\n\nThey're a form of content too, and they have a very clear quality signal: does this issue give a maintainer enough information to act on it, or not? That's binary and measurable. I could build a scoring system around it.\n\nThe result was **Signal-OSS** — a GitHub Action that scores every new issue from 0 to 10, always posts a structured comment with a checklist tailored to that score, and edits that comment if the contributor updates their issue.\n\nI placed 5th overall and won the **Best Detection Accuracy Award**. Here's what I actually built.\n\nMost repos handle bad issues in one of two ways:\n\nNeither approach tells the contributor *what* is actually missing. That's the gap Signal-OSS targets.\n\nThe goal isn't to reject bad issues — it's to convert them into good ones by being specific about what's missing.\n\nWhen a new issue is opened, Signal-OSS runs and always posts a comment — within seconds. What changes based on the score is the *content* of that comment: a high-scoring issue gets a checklist confirming what's already present, while a low-scoring one gets a checklist of what's specifically missing from that issue.\n\na newly opened GitHub issue with the bot comment appearing automatically at the bottom\n\nThe comment isn't generic. It's generated based on the actual signals detected in *that* issue — not a boilerplate reply, but a specific breakdown the contributor can act on.\n\nThis is the part I find most interesting to explain.\n\nThe naive approach would be: send the issue to an LLM, ask it to score it, done. That works, but it couples every issue to an external API call — slow, potentially costly, and requires a key just to get started.\n\nSignal-OSS takes a different approach: **heuristics first, LLM only as an optional tiebreaker.**\n\nBefore any LLM is involved, the action checks 7 rule-based signals:\n\nEach signal contributes to the score. Issues that pass most of these signals will likely land above 7. Issues that fail most will land below 3. Both cases get handled without any LLM call.\n\nIssues that land in the 4–6 range are genuinely ambiguous. Maybe the repro steps are there but vague. Maybe there's a stack trace but no environment context. For these, the heuristics can't reliably decide.\n\nThis is the only zone where an LLM *can* get called — and only if you've provided an API key. Signal-OSS is BYOK: bring your own LLM key if you want the smarter scoring, leave it out and the action still works. For the ambiguous zone specifically, omitting the key falls back to the standard heuristic checklist, which is good enough for most repos.\n\nThe result: **the action is fully functional with zero external API keys**. The LLM layer is an opt-in upgrade, not a requirement.\n\nthe bot's comment showing the checklist and`needs-info`\n\nlabel option\n\nthe bot's comment showing the`signal-oss-ignore`\n\nlabel option\n\nThe bot also:\n\n`needs-info`\n\nlabel automatically on low scores`signal-oss-ignore`\n\nlabel option — if a maintainer adds this to any issue, the bot stays completely silent for that issueThis was a deliberate design constraint I set early: the activation path has to be a single YAML file. No secrets to configure, no dashboard to sign up for.\n\n```\n# .github/workflows/triage.yml\nname: Signal-OSS Issue Triage\n\non:\n  issues:\n    types: [opened, reopened]\n\npermissions:\n  contents: read\n  issues: write\n\njobs:\n  triage:\n    runs-on: ubuntu-latest\n    if: github.actor != 'github-actions[bot]'\n    steps:\n      - name: Triage issue\n        uses: Divyesh-5981/signal-oss@v1\n        env:\n          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n          model: 'none'\n```\n\nThe `GITHUB_TOKEN`\n\nis automatically provided by GitHub Actions — no setup needed. The action works fully without the LLM key, using the standard heuristic checklist for all issues including the ambiguous 4–6 zone. If you add a key, ambiguous issues get LLM-refined scoring instead. Drop the file into `.github/workflows/`\n\nand you're done either way.\n\nthe summary view in the Actions tab showing the run result and the score output\n\nI benchmarked Signal-OSS on real issues from three large repositories: `microsoft/vscode`\n\n, `facebook/react`\n\n, and `rust-lang/rust`\n\n.\n\n| Metric | Score |\n|---|---|\n| Precision | 0.66 |\n| Recall | 0.93 |\n| F1 | 0.77 |\n\nThe precision being lower than recall was intentional. Here's the tradeoff:\n\n**High recall** means the system rarely misses a bad issue. If something is genuinely low-quality, Signal-OSS will catch it.\n\n**Lower precision** means it sometimes flags issues that are actually fine. This results in false positives — good issues that get a comment asking for more detail.\n\nFor this use case, that's the right tradeoff. Missing a bad issue (false negative) wastes maintainer time. Flagging a good issue (false positive) is slightly annoying for the contributor but recoverable — they can see the score is borderline and move on, or the maintainer can apply `signal-oss-ignore`\n\n.\n\nIf I'd optimized for precision, I'd be missing bad issues constantly. That defeats the purpose.\n\nA few honest notes from the hackathon:\n\n**The 4–6 threshold was arbitrary.** I chose it based on intuition and tested it during the benchmark, but a configurable threshold would make Signal-OSS more useful across different repo cultures. A high-volume OSS project might want stricter rules than a small internal tool.\n\n**The heuristics are English-biased.** All 7 signal checks were designed for English-language issues. Issues written in other languages will score lower by default, which isn't fair. This is worth fixing before any serious adoption.\n\n**I benchmarked on large repos.** The issue quality distribution in `vscode`\n\nor `react`\n\nis probably different from smaller, less-structured projects. The benchmark numbers may not generalize.\n\nThe repo is linked below. If you maintain an open-source project and issue triage is eating your time, this is worth a look.\n\nTwo things I'd find genuinely useful from you:\n\n👇 Drop your thoughts below. And if this was useful, the ❤️ helps other maintainers find it.\n\n*Built during the Slop Scan hackathon organized by Hackathon Raptors. Repo in the comments.*", "url": "https://wpnews.pro/news/oss-maintainers-lose-hours-to-vague-issues-i-built-a-github-action-to-fix-that", "canonical_source": "https://dev.to/divyesh5981/oss-maintainers-lose-hours-to-vague-issues-i-built-a-github-action-to-fix-that-4kce", "published_at": "2026-06-24 04:33:55+00:00", "updated_at": "2026-06-24 04:43:46.863078+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "natural-language-processing"], "entities": ["Signal-OSS", "GitHub", "Hackathon Raptors", "Slop Scan"], "alternates": {"html": "https://wpnews.pro/news/oss-maintainers-lose-hours-to-vague-issues-i-built-a-github-action-to-fix-that", "markdown": "https://wpnews.pro/news/oss-maintainers-lose-hours-to-vague-issues-i-built-a-github-action-to-fix-that.md", "text": "https://wpnews.pro/news/oss-maintainers-lose-hours-to-vague-issues-i-built-a-github-action-to-fix-that.txt", "jsonld": "https://wpnews.pro/news/oss-maintainers-lose-hours-to-vague-issues-i-built-a-github-action-to-fix-that.jsonld"}}