# From 48M Records to Top 8 Finalists: How We Built an End-to-End Urban Flow AI

> Source: <https://dev.to/inushathathsara/from-48m-records-to-top-8-finalists-how-we-built-an-end-to-end-urban-flow-ai-53gp>
> Published: 2026-09-24 20:50:13+00:00

We have some exciting news to share! 🎉

Our team, **DataMinds**, competed in the prestigious **SLIIT CodeFest Datathon 2026** and proudly **secured a spot as Finalists (Top 8 teams)** in the Urban Flow Analytics Data Challenge!

In this competition, we tackled a massive, real-world urban transit dataset containing **48.6 million trip records across 260+ urban zones**. The challenge demanded much more than just training a model in a Jupyter Notebook: it required building a production-ready, mathematically sound, and business-viable end-to-end data platform covering everything from chunked streaming data pipelines and spatio-temporal forecasting to an AI-powered executive platform.

Here is the complete behind-the-scenes breakdown of our architectural strategy, the data engineering traps we avoided, our modeling breakthroughs, and the engineering principles that helped us reach the Top 8.

If you prefer a visual walkthrough, check out our official demo video demonstrating the full pipeline, interactive dashboard, and conversational AI mobility assistant:

*(Link: [https://youtu.be/rUpk545ku88](https://youtu.be/rUpk545ku88))*

Rather than treating the challenge as isolated competition tasks, we engineered a cohesive **4-Tier Architecture** that bridges raw streaming telemetry to executive decision-making:

Working with tens of millions of raw geospatial records presents two massive hurdles:

`pd.read_csv()` on 48.6 million rows with 20+ columns will immediately crash standard workstations or cloud instances with Out-Of-Memory (OOM) errors.
To maintain sub-gigabyte RAM footprints during data auditing, we used Python generators with chunked processing (`chunksize=100_000`):

``` python
def stream_audit(csv_path, chunk_size=100_000):
    total_records = 0
    anomalies = {"negative_fare": 0, "excessive_speed": 0, "zero_duration": 0}

    for chunk in pd.read_csv(csv_path, chunksize=chunk_size):
        # Calculate trip duration and implied speed
        duration_hours = (chunk['dropoff_datetime'] - chunk['pickup_datetime']).dt.total_seconds() / 3600.0
        implied_mph = chunk['trip_distance'] / duration_hours.replace(0, np.nan)

        # Track edge cases
        anomalies["negative_fare"] += (chunk['fare_amount'] <= 0).sum()
        anomalies["excessive_speed"] += (implied_mph > 100).sum()
        anomalies["zero_duration"] += (duration_hours <= 0).sum()
        total_records += len(chunk)

    return total_records, anomalies
```

Through our systematic audit across all 48.6M rows, we discovered and programmatically filtered out critical real-world edge cases:

A common mistake in ML hackathons is **data leakage**. If you leak post-trip information into an upfront prediction model, your leaderboard score looks amazing, but your model is completely useless in production.

Random `train_test_split` on time-series mobility data is fatal—it allows models to learn from the future to predict the past. We enforced strict chronological partitioning:

For our Upfront Fare and Duration models, we restricted feature engineering strictly to information available **before the passenger steps into the vehicle**:

`sin(2π * hour / 24)` and `cos(2π * hour / 24)` to preserve diurnal continuity (midnight connects smoothly to 1 AM).
We benchmarked multiple architectures across our holdout test set to select the optimal production model:

| Model Architecture | Fare R² | Fare RMSE ($) | Duration R² | Duration RMSE (min) | Inference Latency | 
|---|---|---|---|---|---|
| Ordinary Least Squares (OLS) | 0.812 | $7.14 | 0.621 | 9.85 min | **0.8 ms** | 
| Random Forest Regressor | 0.941 | $4.12 | 0.789 | 6.94 min | 45.2 ms | 
| **LightGBM Regressor (Winner - Fare)** | **0.9657** | **$3.25** | 0.8142 | 6.55 min | **2.1 ms** | 
| **XGBoost Regressor (Winner - Duration)** | 0.9612 | $3.41 | **0.8268** | **6.39 min** | **3.4 ms** | 

Gradient boosting trees handled the non-linear relationship between Manhattan distance, toll zones, and peak-hour traffic multipliers effortlessly. LightGBM provided lightning-fast inference with sub-cent precision, while XGBoost effectively captured the heavy-tailed variance in urban traffic delays.

Urban mobility is heavily spatial. Having accurate pricing is only half the battle; fleet operators must know **where demand will surge 24, 48, and 72 hours in advance**.

We aggregated zone-level trip demand into hourly buckets and engineered multi-scale temporal lag features:

`t-1`, `t-2`, `t-3` hours.`t-24`, `t-48`, `t-72` hours (same hour over preceding days).`t-168` hours (same day and hour of the previous week).
Our forecaster achieved a test **RMSE of ~1.00 to 1.22 trips/hour** across all major urban zones, allowing dispatchers to pre-position fleet vehicles before surges materialized.

By clustering pickup-to-dropoff vectors across Morning Rush (07:00–10:00), Midday (11:00–14:00), Evening Rush (16:00–19:00), and Late Night (22:00–02:00), we revealed major commercial arterial corridors and airport shuttle dynamics, illuminating severe deadheading (empty return) imbalances.

Judges at modern hackathons don't just want `.ipynb` files; they want to see how engineering impacts business. We translated our models into an **Executive Command Center** built with Streamlit:

We integrated a natural language interface that allows city planners and dispatch managers to ask plain-English questions:

*"What are the top 5 revenue-generating pickup zones during the Friday evening rush?"*

**The Innovation — Ambiguity Guardrails**:

Real users often ask vague questions like *"Show me the best zones"*. Rather than letting the AI hallucinate or make unsafe assumptions, our engine implements strict guardrails that detect ambiguity and clarify whether the user intends "highest volume", "highest fare margin", or "fastest turnaround time".

We mapped model improvements directly to bottom-line business metrics:

If you are competing in data science competitions or datathons, here are four principles that made the difference for Team DataMinds:

`src/data_cleaner.py`, `src/supervised_models.py`, `src/demand_forecaster.py`). This allowed our notebook, test scripts, and UI to share the exact same underlying logic.
Reaching the Top 8 Finalist stage among brilliant teams across the country is an incredible milestone for **Team DataMinds**. 

A huge thank you to the **SLIIT CodeFest Datathon 2026** organizers and judges for organizing such an inspiring, high-impact data challenge.

*Are you building with large-scale mobility data or competing in data challenges? Drop your thoughts or questions in the comments below!*