“Millions of stray animals suffer on the streets or are euthanized in shelters every day around the world.”
That is the opening line of the PetFinder.my Adoption Prediction competition on Kaggle, which released the dataset this project is built on. PetFinder.my has been Malaysia’s leading animal welfare platform since 2008 and holds records for more than 150,000 animals. The premise is that adoption speed is influenced by how a listing presents the animal: its description, attributes, and photos. Shelter capacity is finite, which is what links how long a pet stays listed to the outcome in that opening sentence.
The competition’s stated goal went further than the prediction. Winning models were meant to become tools that guide shelters and rescuers toward listings that get answered.
A rescue volunteer uploads a few photos, writes a short description, fills in a handful of details, and gets back an adoption-speed prediction. That is the competition task, and it stops one step short of the goal. Knowing when a pet is likely to be adopted says nothing about what to change to make it happen sooner.
Adoption Accelerator takes that step. A multimodal pipeline reads the tabular attributes, the description, and the photos, predicts adoption speed with a gradient-boosted ensemble, and passes the prediction and its SHAP attributions to a LangGraph agentic layer, where a vision model inspects the images directly.
What distinguishes the system is what happens before any advice reaches the user. When an agent proposes a change, whether that is a fourth photo or a rewritten description, it does not estimate the effect. It rebuilds the feature vector with the change applied, re-runs the ensemble, and reports the measured delta. Every recommendation therefore carries a number the model produced, and proposals that fail to move the prediction are rejected by the same mechanism that generated them.
Adoption Accelerator predicts how quickly a pet listed on PetFinder.my will be adopted. The model is a soft-voting ensemble of gradient-boosted trees over a single feature matrix built from all three modalities, and it returns a probability distribution across five adoption-speed classes plus SHAP attributions.
On top of that sits a LangGraph multi-agent system I call the Evidence Board. It looks at the photos with a vision model, translates the SHAP drivers into plain language, tests proposed improvements against the real ensemble, and writes a rewritten listing.
One rule governs the whole design: classical ML owns the number, the LLM owns the language, and the LLM never invents a number. Enforcing that third clause took most of the project.
The PetFinder dataset has 14,993 training rows and five classes: same-day, within a week, within a month, within three months, and still listed after a hundred days.
Those classes are ordered, which most classification metrics ignore. If the truth is “still listed after a hundred days” and you predict “within three months,” you are slightly wrong. If you predict “same-day,” you are catastrophically wrong and you have just told a shelter to stop worrying about an animal nobody is asking about. Accuracy scores both mistakes identically.
Quadratic Weighted Kappa does not. It penalizes predictions according to how far they are from the true class (with larger errors penalized quadratically) while accounting for agreement expected by chance. That property does most of the work later, when the model’s probabilities have to be turned into a single label. Before that, the features.
Each modality follows its own extraction pipeline and is merged only at the end, rather than fused inside a single network.
Alongside the learned image embeddings, I extract engineered metadata from the raw images with Google Vision, including object labels, dominant colors, and crop-hint confidence as a proxy for composition. The text branch contributes one additional engineered feature: document-level sentiment. All feature sets are then joined horizontally on PetID.
Late fusion was chosen for two practical reasons. First, tree-based models handle heterogeneous feature groups natively, and each modality stays independently versioned. Second, every feature keeps its modality tag, and the second half of this system depends on that provenance.
To benchmark the system safely, I used Bebé, my neighbor’s two-year-old Maltese. Because he wasn’t actually up for adoption, his listing provided a realistic baseline for running text rewrites against real photos without affecting an animal in need.
For Bebé, text accounted for 71% of the prediction. Four short sentences outweighed the image branch and every tabular attribute combined. That level of attribution is only possible because the modalities were fused late and kept labeled.
Three gradient-boosting libraries went through TPE search with Optuna: LightGBM and XGBoost at thirty trials each, CatBoost at sixteen. The search spaces cover learning rate, depth, regularization, and sampling ratios, with the number of boosting rounds left to native early stopping rather than spent as a search dimension. The production model soft-votes the tuned candidates from all three families over the same 940-feature matrix.
Tuning and ensembling together were worth about +0.045 QWK, lifting a default LightGBM baseline from 0.4488 to 0.4933 for the production ensemble, with the strongest single tuned configuration reaching 0.4979. It set the floor.
What moved the metric sits downstream of the model, in how probabilities become a label. Argmax is the wrong decision rule for an ordinal target. It picks the single most probable class and throws away the ordering that QWK cares about. Instead, compute the expected class value for each sample and cut it into five buckets with four learned boundaries:
That is worth **+0.063 QWK **over argmax, taking the same ensemble from 0.4299 to 0.4933. Everything on the model side, three libraries of hyperparameter search plus the ensemble, bought less than this single change to how the probabilities are turned into a label. On an ordinal target, that is a useful signal about where to spend effort.
Note on validation leakage: optimization thresholds were fitted on validation folds to demonstrate order-of-magnitude impact rather than a leak-free leaderboard metric. In production, cutoffs are fitted strictly out-of-fold.
SHAP splits a single prediction into per-feature contributions that sum back to the score. For tree ensembles, TreeExplainer computes them exactly.
Standard SHAP outputs target human readers with beeswarm plots. Because late fusion preserved the modality tags on every feature, those values can be aggregated by source instead. Across the training set, that yields a stable picture of overall model dependency:
-
Text: 60.8% of mean absolute SHAP mass across 784 features
-
Image: 23.0% across 111 features
-
Tabular: 16.2% across 45 features
Running this same aggregation on a single prediction creates the exact object downstream agents consume. For Bebé, that aggregation is where the 71% text share quoted earlier comes from, against 16% image and 13% tabular, tilted further toward the description than the dataset average.
That instance-level difference is the entire point: the payload describes his specific listing rather than general model behavior. Modalities a listing lacks are omitted entirely, rather than reported as zero.
The per-prediction object is not a visualization. It is typed, small enough to fit cleanly in a prompt, and explains exactly why one specific pet received one specific score. It is the interface between the two halves of the system, and it is what lets the agents make claims backed by real feature attribution.
The Evidence Board is the agentic layer from the title, and it works under one hard constraint: it never retrains the model, re-ranks its outputs, or overrides its predictions. Its only privilege is calling the model and reading what comes back.
The graph runs a deterministic phase first. It preprocesses raw input into the exact feature space the model was trained on, predicts, and then explains. That phase completes in about a second on every run and involves no LLM at all.
Four LLM nodes follow, and three of them are deliberately non-agentic:
-
Visual Analyst. Inspects uploaded photos with a vision model, returns quality scores and observed traits, and flags the strongest photo.
-
Data Analyst. Turns raw SHAP drivers into clear, readable prose using a lightweight model.
-
Synthesizer. Writes the narrative, headline, and rewritten description.
-
Recommendation Agent. The sole autonomous node in the pipeline.
The two analysts run in parallel immediately after inference, and the Synthesizer waits for both. Concentrating agentic behavior into a single node is an intentional design choice rather than a shortcut. Autonomy costs latency and widens the surface area for failure, so it is spent only where the task genuinely requires dynamic decisions. That one node still dominates wall-clock time.
The ReAct Loop and Verification Seam
The recommendation agent runs a bounded ReAct loop: it proposes a hypothesis, calls a tool that rebuilds the feature vector and re-runs the ensemble, reads the measured delta, and chooses what to try next in light of it. Nothing scripts the order of the hypotheses or which features get tested. The loop ends when the agent stops calling tools or when it reaches a budget of eight ensemble runs, whichever comes first.
A separate finalize call then asks for the ranked recommendations, and every item has to cite the *measurement_id *of the run that validated it. Items citing an id that is not in the log are dropped before the report is assembled. That is the seam where the model’s judgment is allowed in and its arithmetic is kept out: the agent chooses and ranks, the measurement log supplies the numbers. If the loop fails outright, a deterministic sweep measures a fixed candidate list instead and the report records that it fell back.
State, Observability, and Operations
The workflow operates on a shared, strongly typed state object, with each node updating only its designated fields. Errors do not abort execution. A failing node appends to a dedicated error list, so the pipeline still produces a valid partial report.
Each execution is fully traced using Langfuse for latency, cost, and debugging. Model choice per node lives in YAML: the Data Analyst runs on a small model, the rest on mid-tier ones. Reassigning any node is a configuration change, not a deployment.
Everything above exists to produce one artifact. The frontend is a Next.js app over a typed backend-for-frontend layer, with types generated from the OpenAPI schema, so a contract change breaks the build rather than the page.
The clearest way to see what the report actually delivers is to run two contrasting cases through it:
Every report opens with a predicted class and a confidence, placed on a spectrum that runs from same-day to never adopted.
Bebé draws adoption within one week at 29.1%. Yuki draws within one month at 35%. Both are moderate-confidence calls with probability spread across neighboring classes, and both reports explicitly say so, rather than rounding the uncertainty away.
The headlines differ in a far more useful way than the classes do:
Neither headline is a language model deciding what sounded important. Both fall directly out of the measurement loop, and the next two panels show how.
Each uploaded photo is scored on sharpness, lighting, framing, and background, with the strongest badged.
Bebé’s photo scores 9 out of 10, with 4s on sharpness, lighting, and framing. Yuki’s scores 7, with 3s on the same three. The prose matches the numbers: Bebé’s photo reads as clear and favourable, while Yuki’s shows a kitten with closed eyes and soft focus.
Yuki’s verdict text also surfaces something the tabular record could not. The record lists a Siamese with a short coat, and the photo does not support either claim. A vision model tuned to flatter the listing would never raise that. This one writes it into the verdict, because the consistency check runs against what the photo actually shows.
This is where text generation is cheap, but validation is costly. When the agent proposes a change, it rebuilds the feature vector, re-runs the ensemble, and reports what moved.
For Yuki, removing the RM 200 fee produces a definitive result: moving the predicted class from one month to one week. That is not a soft probability nudge, it is a hard class shift, measured on the underlying model rather than asserted about it.
Bebé gets the opposite result from the exact same counterfactual machinery:
That second case is instructive: “Charge a small fee to filter for serious adopters” is precisely the kind of plausible-sounding advice an unconstrained language model delivers with full confidence and zero evidence.
An ungrounded model would have told both owners that adding photos significantly speeds up adoption. For Bebé, the measured answer is a shift of a fraction of a percent that leaves his predicted class completely untouched. The system reports the modest result because guessing is not part of its loop.
The last node writes a listing the owner can publish, under a constraint narrower than the measurement loop: the synthesizer may only mention visual traits the vision model actually reported. For Bebé those were a long wavy white coat, round dark eyes, a black nose, floppy furred ears, and a small compact build.
Every visual claim traces directly back to that grounded feature set. Crucially, both rewrites retain the inconvenient facts a human sales pitch would quietly omit:
That last detail follows from a state boundary: even though the system just recommended dropping the fee, the synthesizer still describes the listing as it currently exists. It is architecturally forbidden from writing copy as if a recommended intervention has already been executed.
If every downstream LLM call fails, the pipeline degrades gracefully. The underlying class prediction and SHAP attributions still render, reporting explicitly what was unavailable rather than backfilling missing nodes with plausible text.
Every run carries its own trace, and the report exposes it rather than hiding it behind the narrative.
Fifty-six seconds end to end, eight agent iterations, and less than a cent. The ensemble prediction and the SHAP attributions account for 1.1 seconds of that total, dropping to 0.35 on a warm cache, or under two percent. The recommendation agent takes most of the rest, because every candidate action it tests is another pass through the model.
That split is the whole design in one line of telemetry. Almost none of the time goes to predicting. Almost all of it goes to checking whether the advice is worth giving.
Every result above was produced using the exported dataset from the PetFinder.my Kaggle competition, and an export is a photograph of a system rather than the system itself. Some of what follows is mine. Most of it arrived with the data, and the distinction matters, because the two have very different fixes.
What the competition fixed in place
The target is not a duration. It is five buckets cut out of one: same day, 1 to 7 days, 8 to 30, 31 to 90, and no adoption after 100 days. Those edges came with the dataset. Adopted on day seven and adopted on day eight land one class apart despite being the same event, and the model is penalized for a distinction no shelter would recognize.
The top class is stranger still. Class 4 means no adoption observed after 100 days, which is not “never adopted” but right-censoring wearing a class label. The edges give it away: they run to 90 days and then jump to 100, with nothing recorded in between, a gap sitting in plain sight on the axis of the distribution chart above. Treating that as an ordinal category is the only option the format allows, and it is the wrong statistical object for what the label records.
The accuracy ceiling follows from the same source. QWK around 0.49 is competitive here and modest in absolute terms, because the strongest driver of when a listing gets answered is who was browsing that week, and the export carries no demand signal at all: 940 features, and not one of them records a timestamp, a page view, an inquiry, or an edit. The model is asked to predict a race without being told how many people entered. The train-validation gap has the same origin, with LightGBM reaching 0.999 QWK on train against roughly 0.43 on validation, which is what 940 features over 14,993 rows produce regardless of regularization.
Modifying PhotoAmt inside a feature vector represents an observational feature perturbation, not a causal intervention. The tree ensemble correlates high photo counts with rescuer effort, confounding feature values with user intent.
The measurement is valid inside the feature space; treating it as the effect of a human action is an assumption, and the report is scoped accordingly. The SHAP attributions carry a smaller version of the same caveat, since they are computed on a single LightGBM base learner rather than the full ensemble average.
Read those two lists together and a pattern shows up. Almost nothing on either one is a property of the architecture. They are properties of a static snapshot with the clock stripped out, and they resolve as soon as the same system runs against a platform’s own data.
None of that touches the graph. The ensemble sits behind a tool interface, and the agent measures whatever model is on the other side of it. Swapping a five-class classifier for a hazard model changes one node. The loop that re-runs it, and the rule that no advice ships without a measurement behind it, stay exactly as they are.
The volunteer from the opening still gets a number. What changed is everything attached to it: an explanation of what drove the prediction, a ranked set of actions that were tested rather than suggested, a clear statement when none of them help, and a listing ready to publish the same day.
The contribution here is not the 0.4933. It is the architecture around it, one that lets a generative layer speak freely about language and never about numbers it has not checked. That is what the two reports demonstrate. The same loop that told one owner to drop a fee, and moved Yuki a full class for it, told another that renaming Bebé was worth nothing at all. A system willing to return the disappointing answer is the only kind worth putting in front of someone making a real decision.
Underneath the adoption problem, this was also an experiment in joining two things that usually stay apart: a multimodal machine learning system reading photos, free text, and tabular attributes, and an agentic loop where a generative model has to reason over what that model produced. The disagreements were the interesting part, like a vision model contradicting a record that listed a short-haired Siamese. I would not call the arrangement settled. Exploring multimodal evidence inside an agentic reasoning loop is one of the things I found most promising in this project, and there is still a lot of room to explore where it can go.
The limits are real and they are legible. QWK near 0.49 reflects a label that cannot see who was browsing that week, and the counterfactuals live in feature space rather than in the world. Neither is hidden, and neither is architectural: point the same graph at a platform with timestamps and edit history, and the model behind the tool changes while the loop around it stays where it is.
Which is what makes it worth building. Shelters do not need another number telling them a pet will wait three months. They need to know which of the changes in front of them is worth the hour, and they need to trust the answer enough to act on it, because the listings that sit longest are the ones where that hour counts for most. The prediction itself takes under two percent of the runtime. The rest of this project went into making the advice something a volunteer can defend.
The code is at github.com/PedroMarkovicz/adoption_accelerator
Multimodal Pet Adoption Speed Prediction with an Agentic AI Layer was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.