Predicting Pull Request Acceptance the Moment It's Opened

A first-author paper presented at IEEE COMPSAC 2026 found that pull request acceptance can be predicted from submission-time signals alone, while review effort can only be predicted modestly, using the AIDev corpus of human- and AI-agent-authored PRs. The study evaluated four contributor views — pooled, human-only, agent-only, and balanced — with 5-fold cross-validation stratified by merged/closed label and preprocessing, imputation and hyperparameter tuning confined to the training fold. The authors deliberately excluded post-review features such as comment counts and CI results to avoid leakage, framing the work as binary classification for merge versus close (RQ1) and regression for review comment count and time-to-merge (RQ2).

Predicting Pull Request Acceptance the Moment It's Opened What my IEEE COMPSAC 2026 paper found about predicting PR acceptance and review effort for human and AI-agent pull requests from submission-time signals alone. Every maintainer has a version of the same problem: a queue of open pull requests, limited review time, and no reliable way to tell which ones are close to mergeable and which ones will eat an afternoon. That queue is getting longer now that AI coding agents open pull requests alongside people. In my first-author paper at IEEE COMPSAC 2026 https://ieeexplore.ieee.org/document/11645394 , my co-authors and I asked a narrow question: using only what is known at the moment a PR is opened, can we predict whether it will be accepted, and how much review it will need? The short answer: acceptance, yes, with an important caveat about what “yes” means on an imbalanced dataset. Review effort, only modestly. Both halves of that answer turned out to be useful. The question: why predict a PR’s fate at submission time? Link to section: The question: why predict a PR’s fate at submission time? the-question-why-predict-a-prs-fate-at-submission-time Most prior work on pull request outcomes uses features that only exist after review has started: how many comments a PR has, whether CI passed, how many rounds of changes were requested. Those models score well, but they answer a question nobody needs answered. By the time you know the comment count, you already know how the review is going. The useful moment is earlier. When a PR lands in the queue, a maintainer wants to know two things: 1. Is this likely to be accepted? If so, a quick review might get it merged. If not, it might need a conversation before anyone spends time on line-by-line review. 2. How much effort will this take? A PR that will need a long discussion and a week of back-and-forth should be scheduled differently from one that will merge in an hour. We framed these as two research questions. RQ1 is binary classification: will the PR be merged, or closed without merge? RQ2 is regression: how many review comments will it get, and how long until it merges? Both share one hard constraint: every feature had to be observable when the PR is opened, and nothing else. The data: human and agent PRs in AIDev Link to section: The data: human and agent PRs in AIDev the-data-human-and-agent-prs-in-aidev We built on the AIDev corpus<sup 1 user-content-fn-1 </sup , which contains PR-level metadata, commit-level diffs, review discussions and repository context for pull requests authored by both humans and autonomous coding agents. That mix is what makes the dataset interesting. Agent-authored PRs are new enough that we know very little about how they behave in review, and a model trained only on human PRs might not transfer. To make sure we were not just measuring one population, we evaluated on four contributor views: | View | What it contains | Why it matters | |---|---|---| | Pooled | All PRs, human and agent together | The realistic queue a maintainer sees | | Human-only | PRs authored by people | The baseline most prior work studied | | Agent-only | PRs authored by AI coding agents | Does the model still work on the new population? | | Balanced | The larger group down-sampled so both are equal | Removes the effect of whichever group dominates | Every result below is 5-fold cross-validation, stratified by the merged/closed label for RQ1, with preprocessing, imputation and hyperparameter tuning done only inside the training fold. We compared models with paired tests across folds at . Avoiding leakage: what we deliberately left out Link to section: Avoiding leakage: what we deliberately left out avoiding-leakage-what-we-deliberately-left-out This was the part of the project I spent the most time on, because it is the easiest place to fool yourself. Leakage is when a feature quietly encodes the answer. In PR data it is everywhere: - Comment counts and review timestamps encode how the review went. - Merge timestamps and final-state fields are the label in disguise. - CI results are only known after the pipeline has run, often after reviewers have already reacted. - Later commits and review decisions are literally the outcome of the review. We removed all of these and kept only four groups of features that exist the moment the PR is opened: | Feature group | Examples | |---|---| | PR metadata and text structure | Title length, body length, whether a body exists, whether it contains URLs or fenced code blocks | | Repository and temporal context | Stars, forks, primary language, log-transformed popularity, hour of day and day of week the PR was opened | | Diff structure | Lines added and deleted, files touched, total churn, churn ratio, file-extension diversity, code/docs/test mix | | Task-intent tag | A keyword-rule label from the title and body: fix , feature , refactor , docs or unknown | Notice what is not in that table: no embeddings of the code, no static analysis, no test results, no reviewer-assigned labels. We wanted to know how far cheap, immediately available signals could go before reaching for anything expensive. Results: acceptance is predictable, but read the AUC, not just the F1 Link to section: Results: acceptance is predictable, but read the AUC, not just the F1 results-acceptance-is-predictable-but-read-the-auc-not-just-the-f1 We compared five classical models: Logistic Regression L2, class-balanced , Random Forest 400 trees , Gradient Boosting, Extra Trees, and a small MLP one hidden layer of 128 units, early stopping . There was no reason to reach for anything larger; the feature set is tabular and small, and classical models are easy to inspect. We also included two baselines: a majority-class predictor, and a heuristic that predicts acceptance from body length and number of changed files with thresholds tuned in-fold. Here are the pooled results: | Model | Accuracy | F1 | Precision | Recall | ROC-AUC | |---|---|---|---|---|---| | Random Forest | 0.920 | 0.958 | 0.921 | 0.998 | 0.676 | | Gradient Boosting | 0.919 | 0.958 | 0.920 | 0.998 | 0.675 | | Extra Trees | 0.919 | 0.958 | 0.919 | 1.000 | 0.652 | | MLP | 0.919 | 0.957 | 0.919 | 0.999 | 0.652 | | Majority baseline | 0.918 | 0.957 | 0.918 | 1.000 | 0.500 | | Logistic Regression | 0.772 | 0.866 | 0.936 | 0.807 | 0.617 | | Heuristic baseline | 0.650 | 0.783 | 0.908 | 0.689 | 0.500 | The headline number is that tree-based models reach F1 of 0.958 . But look one row down: the majority baseline, which just says “merged” to everything, scores 0.957. That is because most PRs in AIDev are merged. F1 alone tells you almost nothing here. The column that matters is ROC-AUC . The majority baseline sits at 0.500 because it cannot rank anything. Random Forest and Gradient Boosting reach about 0.68, which means they can separate PRs that will be accepted from PRs that will be closed, using only submission-time signals. That is the real finding: a moderate but real ranking ability that a trivial predictor does not have. It also means these models are not calibrated probability machines. An AUC of 0.68 is enough to sort a review queue. It is not enough to auto-close anything. What the model is actually looking at Link to section: What the model is actually looking at what-the-model-is-actually-looking-at The Random Forest feature importances are strikingly lopsided: Body length is by far the strongest predictor, followed by title length , then repository stars and forks . The diff-size features lines added, lines deleted, files touched, churn barely register for acceptance. In plain terms: how a PR is presented at submission time is strongly associated with whether it gets merged, more than how large the change is. These are predictive signals, not causal claims. A long description does not cause a merge. But it is a reliable marker of the kind of PR that reviewers accept, and if you are building an AI coding agent, that is the cheapest lever you have. The fastest way to improve an agent’s merge rate may not be better code; it may be a better PR description. Does it still work on agent-authored PRs? Link to section: Does it still work on agent-authored PRs? does-it-still-work-on-agent-authored-prs Yes. In the human-only and agent-only views, agent-authored PRs were slightly easier to separate, likely because their formatting and description patterns are more regular. Human PRs vary more in language, structure and scope, so a single decision boundary fits them less cleanly. Both groups stayed predictable. The balanced view was the more interesting check: | Model | Accuracy | F1 | ROC-AUC | |---|---|---|---| | Logistic Regression | 0.873 | 0.932 | 0.571 | | Random Forest | 0.867 | 0.929 | 0.609 | Logistic Regression improved a lot once the contributor groups were balanced, while Random Forest barely moved. The linear model was being pulled around by whichever group dominated the pooled data; the tree ensemble was robust to it. If you only reported pooled numbers you would never see that. Why review effort is much harder to predict Link to section: Why review effort is much harder to predict why-review-effort-is-much-harder-to-predict The second half of the paper was less tidy, and I think it is the more honest contribution. We trained Random Forest and Gradient Boosting regressors on the same submission-time features for two targets: review-comment count an approximate measure of discussion intensity and time-to-merge in hours computed on merged PRs only, since closed PRs have no merge timestamp . | Target | MAE | R² | |---|---|---| | Review comments | 1.00 | 0.20 | | Time-to-merge hours | 24.00 | 0.12 | An of 0.20 for comment count means submission-time features explain about a fifth of the variance in how much discussion a PR gets. For time-to-merge it drops to 0.12, with an average error of a full day. At first that looked like a failure of the features. On reflection, it is a statement about where review effort comes from. Comment count and time-to-merge depend heavily on things that are not properties of the PR at all: - Reviewer availability. The same PR merges in an hour on a quiet Tuesday and in a week during a release freeze. - Project workflow. Some repositories require two approvals and a design discussion; others merge on a single thumbs-up. - CI behavior and queue pressure. A slow pipeline adds hours regardless of what the diff contains. - Team review culture. Some teams comment on every line. Others fix nits themselves after merging. None of these are visible in the PR itself, so no amount of feature engineering on the PR would recover them. One detail I found satisfying: for review effort, the diff features finally mattered . Lines added, lines deleted, files touched and churn were much more prominent for RQ2 than for RQ1. That makes sense. Acceptance is about how a PR is framed and contextualized; effort is about how much reviewers have to read. Two different questions, two different feature families. Agent-authored PRs showed slightly more discussion and a longer median time-to-merge, consistent with reviewers inspecting AI-generated code more cautiously. But prediction errors were comparable across human and agent PRs. Agents change the review dynamics; they do not make the problem unpredictable. What this means for teams using AI coding agents Link to section: What this means for teams using AI coding agents what-this-means-for-teams-using-ai-coding-agents I would draw four conclusions: 1. Early triage is feasible. A model trained on submission-time features can rank a queue by likely acceptance well enough to help a maintainer decide what to look at first. That is a real capability the baselines do not have. 2. The model should advise, not decide. An AUC around 0.68 is a sorting tool, not a gate. Our error analysis found PRs with clear titles, concise descriptions and small diffs that were later rejected for semantic problems or missing tests, and PRs that looked risky many files, vague wording that merged quickly because the change was routine for that repository. An automated accept/reject would get exactly those cases wrong. 3. Treat contributor type as context, not as a quality label. Agent PRs may deserve careful reading, but authorship alone should not mark a PR as risky. The features that predict outcomes are the same for both groups. 4. Effort prediction needs different data. If you want to predict review effort, add reviewer load, author–reviewer history, CI state and project workflow context, not just a richer view of the diff. That is a different dataset and a different paper. For agent developers specifically, the finding on text is actionable today. An agent that writes a clear, well-structured description is producing exactly the signal that reviewers, and models trained on reviewers, respond to. Limitations and what comes next Link to section: Limitations and what comes next limitations-and-what-comes-next Two limitations are worth stating plainly. First, the results are from one corpus. Review practices vary across projects, languages, governance models and CI setups, and the agent-authored PRs in AIDev reflect the agents that existed when the dataset was collected; as coding agents evolve, their PRs will look different. Second, our features are deliberately shallow, and the comment-count target depends on how reliably discussion records could be linked to PRs, so it is an approximate measure of review intensity rather than a full account of reviewer workload. The obvious next step is to add code-aware signals, such as semantic representations of the diff, alongside reviewer and workflow context, and see how much each adds on top of text and metadata. The whole pipeline is deterministic fixed seeds, cached feature tables, preserved fold assignments , and the replication package https://zenodo.org/records/20468479 has the data and scripts if you want to try. Read the paper Link to section: Read the paper read-the-paper - IEEE Xplore: ieeexplore.ieee.org/document/11645394 https://ieeexplore.ieee.org/document/11645394 - arXiv open access : arxiv.org/abs/2607.12057 https://arxiv.org/abs/2607.12057 - Replication package: zenodo.org/records/20468479 https://zenodo.org/records/20468479 - Project page: PR Acceptance Predictor https://kartikpansuriya.com/ project-pr-acceptance-predictor on this site If you work on code review tooling or AI coding agents and want to compare notes, my contact details https://kartikpansuriya.com/ contact are on the homepage. Footnotes Link to section: Footnotes footnote-label 1. H. Li, H. Zhang, and A. E. Hassan. The rise of AI teammates in software engineering SE 3.0: How autonomous coding agents are reshaping software engineering. arXiv:2507.15003 https://arxiv.org/abs/2507.15003 , 2025. ↩ user-content-fnref-1