From 48M Records to Top 8 Finalists: How We Built an End-to-End Urban Flow AI

The DataMinds team built an end-to-end urban flow analytics platform for the SLIIT CodeFest Datathon 2026, processing 48.6 million trip records across 260+ urban zones and finishing as Top 8 finalists. The pipeline used chunked streaming with Python generators to avoid out-of-memory errors, enforced chronological train/test splits to prevent data leakage, and benchmarked OLS, Random Forest, LightGBM and XGBoost models, with LightGBM winning on fare prediction (R² 0.9657, RMSE $3.25) and XGBoost on duration (R² 0.8268, RMSE 6.39 min).

We have some exciting news to share 🎉 Our team, DataMinds , competed in the prestigious SLIIT CodeFest Datathon 2026 and proudly secured a spot as Finalists Top 8 teams in the Urban Flow Analytics Data Challenge In this competition, we tackled a massive, real-world urban transit dataset containing 48.6 million trip records across 260+ urban zones . The challenge demanded much more than just training a model in a Jupyter Notebook: it required building a production-ready, mathematically sound, and business-viable end-to-end data platform covering everything from chunked streaming data pipelines and spatio-temporal forecasting to an AI-powered executive platform. Here is the complete behind-the-scenes breakdown of our architectural strategy, the data engineering traps we avoided, our modeling breakthroughs, and the engineering principles that helped us reach the Top 8. If you prefer a visual walkthrough, check out our official demo video demonstrating the full pipeline, interactive dashboard, and conversational AI mobility assistant: Link: https://youtu.be/rUpk545ku88 https://youtu.be/rUpk545ku88 Rather than treating the challenge as isolated competition tasks, we engineered a cohesive 4-Tier Architecture that bridges raw streaming telemetry to executive decision-making: Working with tens of millions of raw geospatial records presents two massive hurdles: pd.read csv on 48.6 million rows with 20+ columns will immediately crash standard workstations or cloud instances with Out-Of-Memory OOM errors. To maintain sub-gigabyte RAM footprints during data auditing, we used Python generators with chunked processing chunksize=100 000 : python def stream audit csv path, chunk size=100 000 : total records = 0 anomalies = {"negative fare": 0, "excessive speed": 0, "zero duration": 0} for chunk in pd.read csv csv path, chunksize=chunk size : Calculate trip duration and implied speed duration hours = chunk 'dropoff datetime' - chunk 'pickup datetime' .dt.total seconds / 3600.0 implied mph = chunk 'trip distance' / duration hours.replace 0, np.nan Track edge cases anomalies "negative fare" += chunk 'fare amount' <= 0 .sum anomalies "excessive speed" += implied mph 100 .sum anomalies "zero duration" += duration hours <= 0 .sum total records += len chunk return total records, anomalies Through our systematic audit across all 48.6M rows, we discovered and programmatically filtered out critical real-world edge cases: A common mistake in ML hackathons is data leakage . If you leak post-trip information into an upfront prediction model, your leaderboard score looks amazing, but your model is completely useless in production. Random train test split on time-series mobility data is fatal—it allows models to learn from the future to predict the past. We enforced strict chronological partitioning: For our Upfront Fare and Duration models, we restricted feature engineering strictly to information available before the passenger steps into the vehicle : sin 2π hour / 24 and cos 2π hour / 24 to preserve diurnal continuity midnight connects smoothly to 1 AM . We benchmarked multiple architectures across our holdout test set to select the optimal production model: | Model Architecture | Fare R² | Fare RMSE $ | Duration R² | Duration RMSE min | Inference Latency | |---|---|---|---|---|---| | Ordinary Least Squares OLS | 0.812 | $7.14 | 0.621 | 9.85 min | 0.8 ms | | Random Forest Regressor | 0.941 | $4.12 | 0.789 | 6.94 min | 45.2 ms | | LightGBM Regressor Winner - Fare | 0.9657 | $3.25 | 0.8142 | 6.55 min | 2.1 ms | | XGBoost Regressor Winner - Duration | 0.9612 | $3.41 | 0.8268 | 6.39 min | 3.4 ms | Gradient boosting trees handled the non-linear relationship between Manhattan distance, toll zones, and peak-hour traffic multipliers effortlessly. LightGBM provided lightning-fast inference with sub-cent precision, while XGBoost effectively captured the heavy-tailed variance in urban traffic delays. Urban mobility is heavily spatial. Having accurate pricing is only half the battle; fleet operators must know where demand will surge 24, 48, and 72 hours in advance . We aggregated zone-level trip demand into hourly buckets and engineered multi-scale temporal lag features: t-1 , t-2 , t-3 hours. t-24 , t-48 , t-72 hours same hour over preceding days . t-168 hours same day and hour of the previous week . Our forecaster achieved a test RMSE of ~1.00 to 1.22 trips/hour across all major urban zones, allowing dispatchers to pre-position fleet vehicles before surges materialized. By clustering pickup-to-dropoff vectors across Morning Rush 07:00–10:00 , Midday 11:00–14:00 , Evening Rush 16:00–19:00 , and Late Night 22:00–02:00 , we revealed major commercial arterial corridors and airport shuttle dynamics, illuminating severe deadheading empty return imbalances. Judges at modern hackathons don't just want .ipynb files; they want to see how engineering impacts business. We translated our models into an Executive Command Center built with Streamlit: We integrated a natural language interface that allows city planners and dispatch managers to ask plain-English questions: "What are the top 5 revenue-generating pickup zones during the Friday evening rush?" The Innovation — Ambiguity Guardrails : Real users often ask vague questions like "Show me the best zones" . Rather than letting the AI hallucinate or make unsafe assumptions, our engine implements strict guardrails that detect ambiguity and clarify whether the user intends "highest volume", "highest fare margin", or "fastest turnaround time". We mapped model improvements directly to bottom-line business metrics: If you are competing in data science competitions or datathons, here are four principles that made the difference for Team DataMinds: src/data cleaner.py , src/supervised models.py , src/demand forecaster.py . This allowed our notebook, test scripts, and UI to share the exact same underlying logic. Reaching the Top 8 Finalist stage among brilliant teams across the country is an incredible milestone for Team DataMinds . A huge thank you to the SLIIT CodeFest Datathon 2026 organizers and judges for organizing such an inspiring, high-impact data challenge. Are you building with large-scale mobility data or competing in data challenges? Drop your thoughts or questions in the comments below