{"slug": "predicting-pull-request-acceptance-the-moment-it-s-opened", "title": "Predicting Pull Request Acceptance the Moment It's Opened", "summary": "A first-author paper presented at IEEE COMPSAC 2026 found that pull request acceptance can be predicted from submission-time signals alone, while review effort can only be predicted modestly, using the AIDev corpus of human- and AI-agent-authored PRs. The study evaluated four contributor views — pooled, human-only, agent-only, and balanced — with 5-fold cross-validation stratified by merged/closed label and preprocessing, imputation and hyperparameter tuning confined to the training fold. The authors deliberately excluded post-review features such as comment counts and CI results to avoid leakage, framing the work as binary classification for merge versus close (RQ1) and regression for review comment count and time-to-merge (RQ2).", "body_md": "# Predicting Pull Request Acceptance the Moment It's Opened\n\nWhat my IEEE COMPSAC 2026 paper found about predicting PR acceptance and review effort for human and AI-agent pull requests from submission-time signals alone.\n\nEvery maintainer has a version of the same problem: a queue of open pull requests, limited review time, and no reliable way to tell which ones are close to mergeable and which ones will eat an afternoon. That queue is getting longer now that AI coding agents open pull requests alongside people. In my first-author paper at [IEEE COMPSAC 2026](https://ieeexplore.ieee.org/document/11645394), my co-authors and I asked a narrow question: **using only what is known at the moment a PR is opened, can we predict whether it will be accepted, and how much review it will need?**\n\nThe short answer: acceptance, yes, with an important caveat about what “yes” means on an imbalanced dataset. Review effort, only modestly. Both halves of that answer turned out to be useful.\n\n## The question: why predict a PR’s fate at submission time?[Link to section: The question: why predict a PR’s fate at submission time?](#the-question-why-predict-a-prs-fate-at-submission-time)\n\nMost prior work on pull request outcomes uses features that only exist *after* review has started: how many comments a PR has, whether CI passed, how many rounds of changes were requested. Those models score well, but they answer a question nobody needs answered. By the time you know the comment count, you already know how the review is going.\n\nThe useful moment is earlier. When a PR lands in the queue, a maintainer wants to know two things:\n\n1. **Is this likely to be accepted?** If so, a quick review might get it merged. If not, it might need a conversation before anyone spends time on line-by-line review.\n2. **How much effort will this take?** A PR that will need a long discussion and a week of back-and-forth should be scheduled differently from one that will merge in an hour.\n\nWe framed these as two research questions. **RQ1** is binary classification: will the PR be merged, or closed without merge? **RQ2** is regression: how many review comments will it get, and how long until it merges? Both share one hard constraint: every feature had to be observable when the PR is opened, and nothing else.\n\n## The data: human and agent PRs in AIDev[Link to section: The data: human and agent PRs in AIDev](#the-data-human-and-agent-prs-in-aidev)\n\nWe built on the **AIDev** corpus<sup>[1](#user-content-fn-1)</sup>, which contains PR-level metadata, commit-level diffs, review discussions and repository context for pull requests authored by both humans and autonomous coding agents. That mix is what makes the dataset interesting. Agent-authored PRs are new enough that we know very little about how they behave in review, and a model trained only on human PRs might not transfer.\n\nTo make sure we were not just measuring one population, we evaluated on four contributor views:\n\n| View | What it contains | Why it matters | \n|---|---|---|\n| Pooled | All PRs, human and agent together | The realistic queue a maintainer sees | \n| Human-only | PRs authored by people | The baseline most prior work studied | \n| Agent-only | PRs authored by AI coding agents | Does the model still work on the new population? | \n| Balanced | The larger group down-sampled so both are equal | Removes the effect of whichever group dominates | \n\nEvery result below is 5-fold cross-validation, stratified by the merged/closed label for RQ1, with preprocessing, imputation and hyperparameter tuning done only inside the training fold. We compared models with paired tests across folds at .\n\n## Avoiding leakage: what we deliberately left out[Link to section: Avoiding leakage: what we deliberately left out](#avoiding-leakage-what-we-deliberately-left-out)\n\nThis was the part of the project I spent the most time on, because it is the easiest place to fool yourself. Leakage is when a feature quietly encodes the answer. In PR data it is everywhere:\n\n- **Comment counts and review timestamps** encode how the review went.\n- **Merge timestamps and final-state fields** are the label in disguise.\n- **CI results** are only known after the pipeline has run, often after reviewers have already reacted.\n- **Later commits and review decisions** are literally the outcome of the review.\n\nWe removed all of these and kept only four groups of features that exist the moment the PR is opened:\n\n| Feature group | Examples | \n|---|---|\n| PR metadata and text structure | Title length, body length, whether a body exists, whether it contains URLs or fenced code blocks | \n| Repository and temporal context | Stars, forks, primary language, log-transformed popularity, hour of day and day of week the PR was opened | \n| Diff structure | Lines added and deleted, files touched, total churn, churn ratio, file-extension diversity, code/docs/test mix | \n| Task-intent tag | A keyword-rule label from the title and body: `fix` ,`feature` ,`refactor` ,`docs` or`unknown` | \n\nNotice what is *not* in that table: no embeddings of the code, no static analysis, no test results, no reviewer-assigned labels. We wanted to know how far cheap, immediately available signals could go before reaching for anything expensive.\n\n## Results: acceptance is predictable, but read the AUC, not just the F1[Link to section: Results: acceptance is predictable, but read the AUC, not just the F1](#results-acceptance-is-predictable-but-read-the-auc-not-just-the-f1)\n\nWe compared five classical models: Logistic Regression (L2, class-balanced), Random Forest (400 trees), Gradient Boosting, Extra Trees, and a small MLP (one hidden layer of 128 units, early stopping). There was no reason to reach for anything larger; the feature set is tabular and small, and classical models are easy to inspect. We also included two baselines: a **majority-class** predictor, and a **heuristic** that predicts acceptance from body length and number of changed files with thresholds tuned in-fold.\n\nHere are the pooled results:\n\n| Model | Accuracy | F1 | Precision | Recall | ROC-AUC | \n|---|---|---|---|---|---|\n| Random Forest | 0.920 | **0.958** | 0.921 | 0.998 | **0.676** | \n| Gradient Boosting | 0.919 | 0.958 | 0.920 | 0.998 | 0.675 | \n| Extra Trees | 0.919 | 0.958 | 0.919 | 1.000 | 0.652 | \n| MLP | 0.919 | 0.957 | 0.919 | 0.999 | 0.652 | \n| Majority baseline | 0.918 | 0.957 | 0.918 | 1.000 | 0.500 | \n| Logistic Regression | 0.772 | 0.866 | 0.936 | 0.807 | 0.617 | \n| Heuristic baseline | 0.650 | 0.783 | 0.908 | 0.689 | 0.500 | \n\nThe headline number is that tree-based models reach **F1 of 0.958**. But look one row down: the majority baseline, which just says “merged” to everything, scores 0.957. That is because most PRs in AIDev are merged. F1 alone tells you almost nothing here.\n\nThe column that matters is **ROC-AUC**. The majority baseline sits at 0.500 because it cannot rank anything. Random Forest and Gradient Boosting reach about 0.68, which means they *can* separate PRs that will be accepted from PRs that will be closed, using only submission-time signals. That is the real finding: a moderate but real ranking ability that a trivial predictor does not have.\n\nIt also means these models are **not** calibrated probability machines. An AUC of 0.68 is enough to sort a review queue. It is not enough to auto-close anything.\n\n### What the model is actually looking at[Link to section: What the model is actually looking at](#what-the-model-is-actually-looking-at)\n\nThe Random Forest feature importances are strikingly lopsided:\n\n**Body length** is by far the strongest predictor, followed by **title length**, then repository **stars** and **forks**. The diff-size features (lines added, lines deleted, files touched, churn) barely register for acceptance. In plain terms: how a PR is *presented* at submission time is strongly associated with whether it gets merged, more than how large the change is.\n\nThese are predictive signals, not causal claims. A long description does not cause a merge. But it is a reliable marker of the kind of PR that reviewers accept, and if you are building an AI coding agent, that is the cheapest lever you have. The fastest way to improve an agent’s merge rate may not be better code; it may be a better PR description.\n\n### Does it still work on agent-authored PRs?[Link to section: Does it still work on agent-authored PRs?](#does-it-still-work-on-agent-authored-prs)\n\nYes. In the human-only and agent-only views, agent-authored PRs were slightly *easier* to separate, likely because their formatting and description patterns are more regular. Human PRs vary more in language, structure and scope, so a single decision boundary fits them less cleanly. Both groups stayed predictable.\n\nThe balanced view was the more interesting check:\n\n| Model | Accuracy | F1 | ROC-AUC | \n|---|---|---|---|\n| Logistic Regression | 0.873 | 0.932 | 0.571 | \n| Random Forest | 0.867 | 0.929 | 0.609 | \n\nLogistic Regression improved a lot once the contributor groups were balanced, while Random Forest barely moved. The linear model was being pulled around by whichever group dominated the pooled data; the tree ensemble was robust to it. If you only reported pooled numbers you would never see that.\n\n## Why review effort is much harder to predict[Link to section: Why review effort is much harder to predict](#why-review-effort-is-much-harder-to-predict)\n\nThe second half of the paper was less tidy, and I think it is the more honest contribution.\n\nWe trained Random Forest and Gradient Boosting regressors on the same submission-time features for two targets: **review-comment count** (an approximate measure of discussion intensity) and **time-to-merge** in hours (computed on merged PRs only, since closed PRs have no merge timestamp).\n\n| Target | MAE | R² | \n|---|---|---|\n| Review comments | 1.00 | 0.20 | \n| Time-to-merge (hours) | 24.00 | 0.12 | \n\nAn of 0.20 for comment count means submission-time features explain about a fifth of the variance in how much discussion a PR gets. For time-to-merge it drops to 0.12, with an average error of a full day.\n\nAt first that looked like a failure of the features. On reflection, it is a statement about where review effort comes from. Comment count and time-to-merge depend heavily on things that are not properties of the PR at all:\n\n- **Reviewer availability.** The same PR merges in an hour on a quiet Tuesday and in a week during a release freeze.\n- **Project workflow.** Some repositories require two approvals and a design discussion; others merge on a single thumbs-up.\n- **CI behavior and queue pressure.** A slow pipeline adds hours regardless of what the diff contains.\n- **Team review culture.** Some teams comment on every line. Others fix nits themselves after merging.\n\nNone of these are visible in the PR itself, so no amount of feature engineering on the PR would recover them.\n\nOne detail I found satisfying: for review effort, the **diff features finally mattered**. Lines added, lines deleted, files touched and churn were much more prominent for RQ2 than for RQ1. That makes sense. Acceptance is about how a PR is framed and contextualized; effort is about how much reviewers have to read. Two different questions, two different feature families.\n\nAgent-authored PRs showed slightly more discussion and a longer median time-to-merge, consistent with reviewers inspecting AI-generated code more cautiously. But prediction errors were comparable across human and agent PRs. Agents change the review dynamics; they do not make the problem unpredictable.\n\n## What this means for teams using AI coding agents[Link to section: What this means for teams using AI coding agents](#what-this-means-for-teams-using-ai-coding-agents)\n\nI would draw four conclusions:\n\n1. **Early triage is feasible.** A model trained on submission-time features can rank a queue by likely acceptance well enough to help a maintainer decide what to look at first. That is a real capability the baselines do not have.\n2. **The model should advise, not decide.** An AUC around 0.68 is a sorting tool, not a gate. Our error analysis found PRs with clear titles, concise descriptions and small diffs that were later rejected for semantic problems or missing tests, and PRs that looked risky (many files, vague wording) that merged quickly because the change was routine for that repository. An automated accept/reject would get exactly those cases wrong.\n3. **Treat contributor type as context, not as a quality label.** Agent PRs may deserve careful reading, but authorship alone should not mark a PR as risky. The features that predict outcomes are the same for both groups.\n4. **Effort prediction needs different data.** If you want to predict review effort, add reviewer load, author–reviewer history, CI state and project workflow context, not just a richer view of the diff. That is a different dataset and a different paper.\n\nFor agent developers specifically, the finding on text is actionable today. An agent that writes a clear, well-structured description is producing exactly the signal that reviewers, and models trained on reviewers, respond to.\n\n## Limitations and what comes next[Link to section: Limitations and what comes next](#limitations-and-what-comes-next)\n\nTwo limitations are worth stating plainly. First, the results are from one corpus. Review practices vary across projects, languages, governance models and CI setups, and the agent-authored PRs in AIDev reflect the agents that existed when the dataset was collected; as coding agents evolve, their PRs will look different. Second, our features are deliberately shallow, and the comment-count target depends on how reliably discussion records could be linked to PRs, so it is an approximate measure of review intensity rather than a full account of reviewer workload.\n\nThe obvious next step is to add code-aware signals, such as semantic representations of the diff, alongside reviewer and workflow context, and see how much each adds on top of text and metadata. The whole pipeline is deterministic (fixed seeds, cached feature tables, preserved fold assignments), and the [replication package](https://zenodo.org/records/20468479) has the data and scripts if you want to try.\n\n## Read the paper[Link to section: Read the paper](#read-the-paper)\n\n- **IEEE Xplore:**[ieeexplore.ieee.org/document/11645394](https://ieeexplore.ieee.org/document/11645394)\n- **arXiv (open access):**[arxiv.org/abs/2607.12057](https://arxiv.org/abs/2607.12057)\n- **Replication package:**[zenodo.org/records/20468479](https://zenodo.org/records/20468479)\n- **Project page:**[PR Acceptance Predictor](https://kartikpansuriya.com/#project-pr-acceptance-predictor) on this site\n\nIf you work on code review tooling or AI coding agents and want to compare notes, my [contact details](https://kartikpansuriya.com/#contact) are on the homepage.\n\n## Footnotes[Link to section: Footnotes](#footnote-label)\n\n1. \nH. Li, H. Zhang, and A. E. Hassan. *The rise of AI teammates in software engineering (SE) 3.0: How autonomous coding agents are reshaping software engineering.*[arXiv:2507.15003](https://arxiv.org/abs/2507.15003) , 2025.[↩](#user-content-fnref-1)", "url": "https://wpnews.pro/news/predicting-pull-request-acceptance-the-moment-it-s-opened", "canonical_source": "https://kartikpansuriya.com/blog/predicting-pr-acceptance", "published_at": "2026-09-22 00:00:00+00:00", "updated_at": "2026-09-25 02:59:30.364576+00:00", "lang": "en", "topics": ["machine-learning", "ai-agents", "ai-research", "developer-tools"], "entities": ["IEEE COMPSAC 2026", "AIDev"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/predicting-pull-request-acceptance-the-moment-it-s-opened", "markdown": "https://wpnews.pro/news/predicting-pull-request-acceptance-the-moment-it-s-opened.md", "text": "https://wpnews.pro/news/predicting-pull-request-acceptance-the-moment-it-s-opened.txt", "jsonld": "https://wpnews.pro/news/predicting-pull-request-acceptance-the-moment-it-s-opened.jsonld"}}