What I Built
Crawaler is an open-source, AI-powered price comparison platform for Pakistani e-commerce.
It searches products across Daraz and Telemart, groups identical products together, and displays prices side by side so users can instantly find the best deal without opening multiple tabs.
The platform also supports AI image search. Users can upload a product photo, and Google Gemini identifies the product and automatically performs the search across supported stores.
🔗 Live Link: https://crawaler-front-v2.vercel.app
| Layer | Technologies |
|---|---|
| Frontend | React, Vite, TypeScript, Tailwind CSS, shadcn/ui |
| Backend | Node.js, Express, Puppeteer, Cheerio |
| Database | MongoDB, Mongoose |
| AI | Google Gemini |
| Authentication | JWT, bcrypt, Nodemailer |
Online buyers often spend a lot of time checking different websites to compare prices. Crawaler solves this problem by bringing everything into a single platform.
This project started as a side project that remained unfinished for months.
When I returned to it, the Daraz integration was already working, but a comparison platform needs multiple sources to provide real value. Finding and integrating a reliable second platform became the biggest challenge.
I first experimented with OLX and managed to extract some data, but maintaining a stable integration proved difficult. Eventually, I decided to move on.
Next, I explored Temu. Their APIs and requests were heavily protected and obfuscated, making reliable integration extremely challenging. After several attempts, I abandoned this approach.
Finally, I turned to Telemart. While it exposed API, access tokens expired frequently, causing the scraper to fail during execution.
To solve this, I implemented automatic token refresh handling, allowing Crawaler to detect expired tokens and recover seamlessly. This became the reliable second data source the platform needed.
With comparison functionality complete, I added the feature I had wanted from the beginning: AI-powered image search.
Buyers can upload a product image, Google Gemini identifies the product, and Crawaler automatically searches and compares prices across supported platforms.
GitHub Copilot played an important role in helping me finish the project.
It assisted with debugging scraper issues, improving reliability, generating boilerplate code, refining React components, and accelerating development when I was stuck on implementation details.
Copilot helped me focus more on solving complex engineering challenges while spending less time on repetitive tasks. It was especially useful while refining scraping logic, handling token refresh workflows, and improving overall code quality.
Crawaler transformed from an unfinished side project into a fully functional platform that helps buyers save both time and money.
The GitHub Finish-Up-A-Thon Challenge provided the motivation I needed to finally complete something that had been sitting unfinished for months. I'm excited to continue improving the platform, add more store integrations, and welcome contributions from the open-source community.