{"slug": "from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai", "title": "From 48M Records to Top 8 Finalists: How We Built an End-to-End Urban Flow AI", "summary": "The DataMinds team built an end-to-end urban flow analytics platform for the SLIIT CodeFest Datathon 2026, processing 48.6 million trip records across 260+ urban zones and finishing as Top 8 finalists. The pipeline used chunked streaming with Python generators to avoid out-of-memory errors, enforced chronological train/test splits to prevent data leakage, and benchmarked OLS, Random Forest, LightGBM and XGBoost models, with LightGBM winning on fare prediction (R² 0.9657, RMSE $3.25) and XGBoost on duration (R² 0.8268, RMSE 6.39 min).", "body_md": "We have some exciting news to share! 🎉\n\nOur team, **DataMinds**, competed in the prestigious **SLIIT CodeFest Datathon 2026** and proudly **secured a spot as Finalists (Top 8 teams)** in the Urban Flow Analytics Data Challenge!\n\nIn this competition, we tackled a massive, real-world urban transit dataset containing **48.6 million trip records across 260+ urban zones**. The challenge demanded much more than just training a model in a Jupyter Notebook: it required building a production-ready, mathematically sound, and business-viable end-to-end data platform covering everything from chunked streaming data pipelines and spatio-temporal forecasting to an AI-powered executive platform.\n\nHere is the complete behind-the-scenes breakdown of our architectural strategy, the data engineering traps we avoided, our modeling breakthroughs, and the engineering principles that helped us reach the Top 8.\n\nIf you prefer a visual walkthrough, check out our official demo video demonstrating the full pipeline, interactive dashboard, and conversational AI mobility assistant:\n\n*(Link: [https://youtu.be/rUpk545ku88](https://youtu.be/rUpk545ku88))*\n\nRather than treating the challenge as isolated competition tasks, we engineered a cohesive **4-Tier Architecture** that bridges raw streaming telemetry to executive decision-making:\n\nWorking with tens of millions of raw geospatial records presents two massive hurdles:\n\n`pd.read_csv()` on 48.6 million rows with 20+ columns will immediately crash standard workstations or cloud instances with Out-Of-Memory (OOM) errors.\nTo maintain sub-gigabyte RAM footprints during data auditing, we used Python generators with chunked processing (`chunksize=100_000`):\n\n``` python\ndef stream_audit(csv_path, chunk_size=100_000):\n    total_records = 0\n    anomalies = {\"negative_fare\": 0, \"excessive_speed\": 0, \"zero_duration\": 0}\n\n    for chunk in pd.read_csv(csv_path, chunksize=chunk_size):\n        # Calculate trip duration and implied speed\n        duration_hours = (chunk['dropoff_datetime'] - chunk['pickup_datetime']).dt.total_seconds() / 3600.0\n        implied_mph = chunk['trip_distance'] / duration_hours.replace(0, np.nan)\n\n        # Track edge cases\n        anomalies[\"negative_fare\"] += (chunk['fare_amount'] <= 0).sum()\n        anomalies[\"excessive_speed\"] += (implied_mph > 100).sum()\n        anomalies[\"zero_duration\"] += (duration_hours <= 0).sum()\n        total_records += len(chunk)\n\n    return total_records, anomalies\n```\n\nThrough our systematic audit across all 48.6M rows, we discovered and programmatically filtered out critical real-world edge cases:\n\nA common mistake in ML hackathons is **data leakage**. If you leak post-trip information into an upfront prediction model, your leaderboard score looks amazing, but your model is completely useless in production.\n\nRandom `train_test_split` on time-series mobility data is fatal—it allows models to learn from the future to predict the past. We enforced strict chronological partitioning:\n\nFor our Upfront Fare and Duration models, we restricted feature engineering strictly to information available **before the passenger steps into the vehicle**:\n\n`sin(2π * hour / 24)` and `cos(2π * hour / 24)` to preserve diurnal continuity (midnight connects smoothly to 1 AM).\nWe benchmarked multiple architectures across our holdout test set to select the optimal production model:\n\n| Model Architecture | Fare R² | Fare RMSE ($) | Duration R² | Duration RMSE (min) | Inference Latency | \n|---|---|---|---|---|---|\n| Ordinary Least Squares (OLS) | 0.812 | $7.14 | 0.621 | 9.85 min | **0.8 ms** | \n| Random Forest Regressor | 0.941 | $4.12 | 0.789 | 6.94 min | 45.2 ms | \n| **LightGBM Regressor (Winner - Fare)** | **0.9657** | **$3.25** | 0.8142 | 6.55 min | **2.1 ms** | \n| **XGBoost Regressor (Winner - Duration)** | 0.9612 | $3.41 | **0.8268** | **6.39 min** | **3.4 ms** | \n\nGradient boosting trees handled the non-linear relationship between Manhattan distance, toll zones, and peak-hour traffic multipliers effortlessly. LightGBM provided lightning-fast inference with sub-cent precision, while XGBoost effectively captured the heavy-tailed variance in urban traffic delays.\n\nUrban mobility is heavily spatial. Having accurate pricing is only half the battle; fleet operators must know **where demand will surge 24, 48, and 72 hours in advance**.\n\nWe aggregated zone-level trip demand into hourly buckets and engineered multi-scale temporal lag features:\n\n`t-1`, `t-2`, `t-3` hours.`t-24`, `t-48`, `t-72` hours (same hour over preceding days).`t-168` hours (same day and hour of the previous week).\nOur forecaster achieved a test **RMSE of ~1.00 to 1.22 trips/hour** across all major urban zones, allowing dispatchers to pre-position fleet vehicles before surges materialized.\n\nBy clustering pickup-to-dropoff vectors across Morning Rush (07:00–10:00), Midday (11:00–14:00), Evening Rush (16:00–19:00), and Late Night (22:00–02:00), we revealed major commercial arterial corridors and airport shuttle dynamics, illuminating severe deadheading (empty return) imbalances.\n\nJudges at modern hackathons don't just want `.ipynb` files; they want to see how engineering impacts business. We translated our models into an **Executive Command Center** built with Streamlit:\n\nWe integrated a natural language interface that allows city planners and dispatch managers to ask plain-English questions:\n\n*\"What are the top 5 revenue-generating pickup zones during the Friday evening rush?\"*\n\n**The Innovation — Ambiguity Guardrails**:\n\nReal users often ask vague questions like *\"Show me the best zones\"*. Rather than letting the AI hallucinate or make unsafe assumptions, our engine implements strict guardrails that detect ambiguity and clarify whether the user intends \"highest volume\", \"highest fare margin\", or \"fastest turnaround time\".\n\nWe mapped model improvements directly to bottom-line business metrics:\n\nIf you are competing in data science competitions or datathons, here are four principles that made the difference for Team DataMinds:\n\n`src/data_cleaner.py`, `src/supervised_models.py`, `src/demand_forecaster.py`). This allowed our notebook, test scripts, and UI to share the exact same underlying logic.\nReaching the Top 8 Finalist stage among brilliant teams across the country is an incredible milestone for **Team DataMinds**. \n\nA huge thank you to the **SLIIT CodeFest Datathon 2026** organizers and judges for organizing such an inspiring, high-impact data challenge.\n\n*Are you building with large-scale mobility data or competing in data challenges? Drop your thoughts or questions in the comments below!*", "url": "https://wpnews.pro/news/from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai", "canonical_source": "https://dev.to/inushathathsara/from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai-53gp", "published_at": "2026-09-24 20:50:13+00:00", "updated_at": "2026-09-24 20:59:00.704841+00:00", "lang": "en", "topics": ["machine-learning", "ai-tools", "ai-products"], "entities": ["DataMinds", "SLIIT CodeFest Datathon 2026", "LightGBM", "XGBoost", "Random Forest"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai", "markdown": "https://wpnews.pro/news/from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai.md", "text": "https://wpnews.pro/news/from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai.txt", "jsonld": "https://wpnews.pro/news/from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai.jsonld"}}