This is a submission for the MLH x DEV Writing Challenge Smart Connect gives AI agents something they usually don't have: the ability to pick up the phone.
Ask most AI assistants to book you a haircut and they'll give you a phone number. You still have to make the call, wait on hold and go back and forth over times. We wanted an agent that actually does it.
In Smart Connect you create your own agents and choose exactly what each one is allowed to do: phone calls, email, calendar, payments. Then you just ask in plain English:
"Book a dentist appointment tomorrow at 3pm. If that doesn't work, 2 or 2:30 is fine."
The agent plans the call, a voice agent phones the business and has a real conversation, and you get back the transcript, a short summary and the confirmed booking. It only offers the alternatives you allowed, and it never agrees to pay a deposit without your approval.
The biggest thing I learned: with voice agents, less is more. Every time a call sounded robotic, our instinct was to add another rule to the prompt, and every rule made it worse. Short, simple instructions are what made it sound human.
Code: https://github.com/JamilHassiba/oxhack26 What a call looks like in the app:
Gemini runs each agent. We used function calling so it can turn a message into a place_call with the business name, the goal, any constraints, and the alternative times you said were fine.
The tricky part: a phone call takes minutes, so Gemini can't just wait for an answer. We **** the function-calling loop when the call starts, save its state, and resume it when the call ends. Gemini then reads the transcript, extracts a structured result with JSON output (booked? what time? what else was offered?), and decides what to do next. That might be calling back once with an offered time you allowed, adding the event to your calendar, or emailing you a confirmation.
The phone conversation itself is an ElevenLabs Conversational AI agent, placed over Twilio. For every call we pass it dynamic variables: who it's calling for, the goal, the constraints and the allowed alternatives. When the call ends, ElevenLabs sends the full transcript to our backend through a signed post-call webhook (we verify the HMAC signature). We also gave it a server tool for asking the user to approve a deposit mid-call.
What surprised us: ElevenLabs only gives you an agent call's transcript after the call ends. We spent a while building a "live transcript" that polled for it mid-call, before realising the data simply isn't there yet. So we switched to following the call's status through Twilio and built an animated "on the call" view instead.
Our whole database is Tiger Data (Postgres + TimescaleDB). Each call is stored as a time-series of events (dialling, ringing, answered, every transcript line, ended) in a hypertable. The chat streams those events to the browser with server-sent events, and that's what moves the call card from "Calling" to "On the call" in real time. Modelling a phone call as time-series data turned out to be a really natural fit.
We deployed on a Vultr VM in London: FastAPI running under systemd, behind Caddy for HTTPS, which also serves the React frontend. Having a real public URL mattered, because ElevenLabs needs somewhere to send its webhooks.
We built Smart Connect at OxHack '26 in Oxford.
What broke at 3am:
We started as a team of three and finished as two after a teammate had to leave, so we re-planned mid-hackathon and kept going.