{"slug": "an-ai-agent-that-makes-phone-calls-oxhack-26", "title": "An AI agent that makes phone calls (OxHack ‘26)", "summary": "A two-person team at OxHack '26 in Oxford built Smart Connect, an AI agent system that places real phone calls to businesses on a user's behalf, using Gemini function calling to plan calls, ElevenLabs Conversational AI over Twilio to hold the conversation, and a signed post-call webhook to return transcripts and structured booking results. The agent pauses and resumes its function-calling loop across the multi-minute call, stores each call as a time-series of events in a Tiger Data Postgres/TimescaleDB hypertable, and streams status to a React frontend via server-sent events. The developer's main lesson: \"with voice agents, less is more\" — adding prompt rules made calls sound more robotic, while short instructions sounded human.", "body_md": "*This is a submission for the [MLH x DEV Writing Challenge](https://dev.to/mlh-hackathon)*\n\n**Smart Connect** gives AI agents something they usually don't have: the ability to pick up the phone.\n\nAsk most AI assistants to book you a haircut and they'll give you a phone number. You still have to make the call, wait on hold and go back and forth over times. We wanted an agent that actually does it.\n\nIn Smart Connect you create your own agents and choose exactly what each one is allowed to do: phone calls, email, calendar, payments. Then you just ask in plain English:\n\n\"Book a dentist appointment tomorrow at 3pm. If that doesn't work, 2 or 2:30 is fine.\"\n\nThe agent plans the call, a voice agent phones the business and has a real conversation, and you get back the transcript, a short summary and the confirmed booking. It only offers the alternatives you allowed, and it never agrees to pay a deposit without your approval.\n\nThe biggest thing I learned: **with voice agents, less is more.** Every time a call sounded robotic, our instinct was to add another rule to the prompt, and every rule made it worse. Short, simple instructions are what made it sound human.\n\nCode: [https://github.com/JamilHassiba/oxhack26](https://github.com/JamilHassiba/oxhack26)\n\nWhat a call looks like in the app:\n\nGemini runs each agent. We used **function calling** so it can turn a message into a `place_call` with the business name, the goal, any constraints, and the alternative times you said were fine.\n\nThe tricky part: a phone call takes minutes, so Gemini can't just wait for an answer. We **pause** the function-calling loop when the call starts, save its state, and **resume** it when the call ends. Gemini then reads the transcript, extracts a structured result with **JSON output** (booked? what time? what else was offered?), and decides what to do next. That might be calling back once with an offered time you allowed, adding the event to your calendar, or emailing you a confirmation.\n\nThe phone conversation itself is an **ElevenLabs Conversational AI** agent, placed over Twilio. For every call we pass it dynamic variables: who it's calling for, the goal, the constraints and the allowed alternatives. When the call ends, ElevenLabs sends the full transcript to our backend through a **signed post-call webhook** (we verify the HMAC signature). We also gave it a server tool for asking the user to approve a deposit mid-call.\n\nWhat surprised us: ElevenLabs only gives you an agent call's transcript **after** the call ends. We spent a while building a \"live transcript\" that polled for it mid-call, before realising the data simply isn't there yet. So we switched to following the call's status through Twilio and built an animated \"on the call\" view instead.\n\nOur whole database is **Tiger Data** (Postgres + TimescaleDB). Each call is stored as a time-series of events (dialling, ringing, answered, every transcript line, ended) in a **hypertable**. The chat streams those events to the browser with server-sent events, and that's what moves the call card from \"Calling\" to \"On the call\" in real time. Modelling a phone call as time-series data turned out to be a really natural fit.\n\nWe deployed on a **Vultr** VM in London: FastAPI running under systemd, behind Caddy for HTTPS, which also serves the React frontend. Having a real public URL mattered, because ElevenLabs needs somewhere to send its webhooks.\n\nWe built Smart Connect at **OxHack '26 in Oxford**.\n\nWhat broke at 3am:\n\nWe started as a team of three and finished as two after a teammate had to leave, so we re-planned mid-hackathon and kept going.", "url": "https://wpnews.pro/news/an-ai-agent-that-makes-phone-calls-oxhack-26", "canonical_source": "https://dev.to/jamilhassiba/an-ai-agent-that-makes-phone-calls-oxhack-26-426k", "published_at": "2026-10-29 11:43:49+00:00", "updated_at": "2026-10-29 11:46:56.764765+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "large-language-models", "generative-ai", "developer-tools"], "entities": ["Smart Connect", "Gemini", "ElevenLabs", "Twilio", "Tiger Data", "Vultr", "OxHack '26", "FastAPI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/an-ai-agent-that-makes-phone-calls-oxhack-26", "markdown": "https://wpnews.pro/news/an-ai-agent-that-makes-phone-calls-oxhack-26.md", "text": "https://wpnews.pro/news/an-ai-agent-that-makes-phone-calls-oxhack-26.txt", "jsonld": "https://wpnews.pro/news/an-ai-agent-that-makes-phone-calls-oxhack-26.jsonld"}}