An AI agent that makes phone calls (OxHack ‘26)

A two-person team at OxHack '26 in Oxford built Smart Connect, an AI agent system that places real phone calls to businesses on a user's behalf, using Gemini function calling to plan calls, ElevenLabs Conversational AI over Twilio to hold the conversation, and a signed post-call webhook to return transcripts and structured booking results. The agent pauses and resumes its function-calling loop across the multi-minute call, stores each call as a time-series of events in a Tiger Data Postgres/TimescaleDB hypertable, and streams status to a React frontend via server-sent events. The developer's main lesson: "with voice agents, less is more" — adding prompt rules made calls sound more robotic, while short instructions sounded human.

This is a submission for the MLH x DEV Writing Challenge https://dev.to/mlh-hackathon Smart Connect gives AI agents something they usually don't have: the ability to pick up the phone. Ask most AI assistants to book you a haircut and they'll give you a phone number. You still have to make the call, wait on hold and go back and forth over times. We wanted an agent that actually does it. In Smart Connect you create your own agents and choose exactly what each one is allowed to do: phone calls, email, calendar, payments. Then you just ask in plain English: "Book a dentist appointment tomorrow at 3pm. If that doesn't work, 2 or 2:30 is fine." The agent plans the call, a voice agent phones the business and has a real conversation, and you get back the transcript, a short summary and the confirmed booking. It only offers the alternatives you allowed, and it never agrees to pay a deposit without your approval. The biggest thing I learned: with voice agents, less is more. Every time a call sounded robotic, our instinct was to add another rule to the prompt, and every rule made it worse. Short, simple instructions are what made it sound human. Code: https://github.com/JamilHassiba/oxhack26 https://github.com/JamilHassiba/oxhack26 What a call looks like in the app: Gemini runs each agent. We used function calling so it can turn a message into a place call with the business name, the goal, any constraints, and the alternative times you said were fine. The tricky part: a phone call takes minutes, so Gemini can't just wait for an answer. We pause the function-calling loop when the call starts, save its state, and resume it when the call ends. Gemini then reads the transcript, extracts a structured result with JSON output booked? what time? what else was offered? , and decides what to do next. That might be calling back once with an offered time you allowed, adding the event to your calendar, or emailing you a confirmation. The phone conversation itself is an ElevenLabs Conversational AI agent, placed over Twilio. For every call we pass it dynamic variables: who it's calling for, the goal, the constraints and the allowed alternatives. When the call ends, ElevenLabs sends the full transcript to our backend through a signed post-call webhook we verify the HMAC signature . We also gave it a server tool for asking the user to approve a deposit mid-call. What surprised us: ElevenLabs only gives you an agent call's transcript after the call ends. We spent a while building a "live transcript" that polled for it mid-call, before realising the data simply isn't there yet. So we switched to following the call's status through Twilio and built an animated "on the call" view instead. Our whole database is Tiger Data Postgres + TimescaleDB . Each call is stored as a time-series of events dialling, ringing, answered, every transcript line, ended in a hypertable . The chat streams those events to the browser with server-sent events, and that's what moves the call card from "Calling" to "On the call" in real time. Modelling a phone call as time-series data turned out to be a really natural fit. We deployed on a Vultr VM in London: FastAPI running under systemd, behind Caddy for HTTPS, which also serves the React frontend. Having a real public URL mattered, because ElevenLabs needs somewhere to send its webhooks. We built Smart Connect at OxHack '26 in Oxford . What broke at 3am: We started as a team of three and finished as two after a teammate had to leave, so we re-planned mid-hackathon and kept going.