Early Lessons on What It Takes to Get a Generative AI-Enabled Medical Device Past FDA

Limbic Inc.'s generative AI-enabled medical device Unpacked, which delivers CBT-based treatment via a voice agent for depression, is on the U.S. market under the FDA's TEMPO pilot, marking the first such device with FDA's explicit knowledge under an enforcement-discretion pilot. The device is not yet FDA authorized but is part of a pilot that defers evidence collection to post-market, with Limbic's peer-reviewed study in Nature Medicine providing the best public evidence. FDA's CDRH has published a discussion paper on regulating generative AI medical devices, and Limbic's study offers a concrete example of the evidence that may be required.

It takes FDA time to adapt to new technology. Even today, I think FDA is less comfortable regulating SaMD than traditional devices. Early submissions that use a new technology tend to move slowly. As FDA gains experience, it gets easier: more reviewers know the technology, and public submissions create useful precedents. We saw this with traditional AI, especially in radiology, where FDA now has a clearer sense of how to classify these devices and what evidence to expect. FDA is still catching up with generative AI. FDA is working to build the regulatory science, and the first few submissions will likely be slow and hard. For those of us working at the edge of the field, it is exciting to take part in this process and work with FDA to find a path forward for bringing this technology to market. I’m therefore thrilled about a new generative AI-enabled device that is on the market in the U.S. The product is Unpacked, from Limbic Inc., and it is worth studying closely if you are building agentic AI for clinical use . It is not FDA authorized yet , but is on the market as part of the TEMPO pilot. I’ve written about this pilot in more detail here https://innolitics.com/articles/dexcom-tempo-staged-evidence-wearable-ai/ , but it is effectively a new, narrowly defined market pathway that pushes off evidence collection into the “post-market.” This real-world performance data will ultimately be used to file a 510 k or De Novo. FDA has thus agreed that it is a medical device and also that there’s sufficient clinical evidence to put it on the market. As far as I can tell, Unpacked is the first generative AI-enabled medical device doing substantial clinical work in the US with FDA’s explicit knowledge under an enforcement-discretion pilot. UpDoc K253281 was cleared last year and uses a voice agent to communicate with patients, but the voice AI has a limited role; you can read more in my partner Yujan’s in-depth analysis https://innolitics.com/articles/updoc-fda-cleared-ai-agent/ . Three other companies are in the TEMPO pilot: All are interesting, but none of them has published much about the study designs behind their products I’ll be writing about future TEMPO participants too, so be sure to subscribe https://innolitics.com/newsletter/ . Limbic has seven peer-reviewed papers, including a randomized, double-blind evaluation of its therapy agents published in Nature Medicine this March. The public record does not tell us exactly what evidence FDA reviewed, but this study is the best public look we have at the evidence behind a device accepted into TEMPO. At about the same time Limbic’s participation in TEMPO was announced, CDRH published a discussion paper, Considerations for the Regulation of Generative AI-Enabled Medical Devices https://www.fda.gov/medical-devices/digital-health-center-excellence/considerations-regulation-generative-ai-enabled-medical-devices-discussion-paper-and-request . The paper is abstract, while Limbic’s study gives us a concrete example. In the rest of this article, I examine the Nature Medicine study alongside FDA’s discussion paper. Together, they offer an early view of the evidence that may be needed if FDA creates rules for agentic AI devices. To understand why Unpacked matters, it helps to start with what the device actually does. As of late August 2026, a Medicare beneficiary with depression can pick up their phone and call an AI voice agent that delivers structured, CBT-based treatment within a clinician-supervised outpatient service Patients receive scheduled twenty-minute telephone sessions—no app, no smartphone required—plus 24/7 check-in availability. Licensed clinicians employed by Limbic's provider entity supervise every patient: they review sessions and outcomes, receive real-time safety alerts, and can adjust or take over treatment. The contraindication list is long: These exclusions tell you how deliberately the severe end of the risk spectrum has been carved away. In terms of FDA’s proposed two-axis risk model, I’d place Limbic under the Action-Taking HCP-Supervised with limited consequences. Without these contraindications it may bump it up to Moderate consequences. Once they get their De Novo, I’m sure Limbic will start with the lower risk indications and will expand from there. I think this is a good strategy for many generative AI-enabled devices. Begin with lower risk indications perhaps even a general wellness device . Gather real-world evidence to show that it works and is safe there, then expand your indications into higher risk territory. Unpacked uses what Limbic calls a cognitive layer: a clinical reasoning and safety layer wrapped around a general-purpose foundation LLM. It operates on both sides of the model. On the input side, it processes each patient message before deciding what to do. It screens for risk, routes an at-risk patient to a human instead of the LLM, infers the patient’s clinical state, and combines the message with clinical context to build a tightly constrained prompt. On the output side, it checks each draft response for safety and clinical validity and verifies that the LLM followed its instructions before the response is spoken to the patient. The foundation model in the middle can be swapped. The input layer and output layer consist of safety and clinical reasoning models a combination of prompt engineering, fine-tuned Llama 3 models, embedding-based classifiers, and decision trees . For example, they use a risk-detection model that flags messages that contain suicidal ideation or the risk of abuse to others. Generative AI’s unique strengths—its flexibility in responding to variable inputs, its adaptability, and its support for multi-step interactive workflows—also make it more difficult to evaluate. First, generative AI has a huge range of possible inputs. Consider a voice AI therapy agent. Each round of interaction with a patient can vary as widely as human speech. Second, these devices are often interactive. Most traditional AI-enabled devices accept an input and produce an output, so a fixed test set with known correct outputs can be reused across software versions. Fixed component benchmarks may also be reused after an update to a generative AI-enabled device. But end-to-end testing is harder: a new model may respond differently, which changes what the user says next. A major update may therefore require new interactive study data. I predict that this will make performance testing of interactive devices more expensive. FDA appears to be aware of these problems and in their discussion paper they propose a three-pronged approach: Limbic’s Nature Medicine paper actually covers all three types of validation. The paper goes into the least detail about their benchmarking. It consists of testing individual models within their device on fixed datasets. Not many details are provided and the dataset sizes appear to include both tuning and testing datasets: | Model Purpose | Dataset Size | Performance | |---|---|---| | Risk detection: Binary classifier of user messages to identify risk of suicidal ideation or abuse to others. | 2,626 | F1=0.69 | | Identifying Sensitive Topics: Multi-class classifier of user messages of 18 sensitive topics e.g., avoiding deep questioning into traumatic experiences | 284 | F1=0.63 | | Jailbreak Detection: Binary classifier of user messages for the detection of jailbreak attempts or off-topic conversation | 546 | F1=0.84 | | Problem List Identifier: Model that analyzes conversation history to identify a list of patient problems, which ones have been explored, and what should be discussed next. | 6,457 | F1=0.75 | | State Inference: Multi-class classification model outputting the presence of 16 CBT-relevant cognitive, behavioral, and emotional states. | 11,371 | F1=0.78 | | Intervention Selection Model: A hand-tuned decision tree based on the CBT manual. | N/A | N/A | | Intervention Delivery Model: Prompt optimized model for gathering source material for the conversational LLM. | 1,364 | Accuracy=96% | | Safety Output Model: Multi-class classifier to detect if output messages are 1 harmful to patients or 2 violating medical device regulations. | 1,400 | F1=0.73 | | Therapy Adherence Model: Binary classifier of the conversational LLM’s output as to if it continues the treatment vs stops the treatment. | 26,000 | F1=0.59 | | Recovery Model: Binary classifier of the conversational LLM’s output as to whether it improves symptoms above average or below average. | 3,400 | F1=0.61 | Note that each of these benchmarks focuses on a particular part of the overall device; this allows the datasets to be re-used as the various components are improved, unlike the clinical competency datasets discussed below, which are tied to a particular version of the model. It is exciting to see some initial sample sizes and performance metrics that other developers can use as a starting point when setting their own acceptance criteria. In an FDA submission, I would expect additional details about how the various datasets were sampled, relevant subgroups, how the reference standard was determined, and what predefined acceptance criteria were used. There are a lot of components in Limbic’s device; it seems clear that all of this additional structure will constrain the conversational LLM’s outputs. This is a tradeoff between giving the model flexibility to adapt to new inputs and keeping its performance predictable. The Nature Medicine paper provides a concrete example of “clinical confirmation”. FDA describes several potential approaches in their discussion paper, ranging from retrospective testing to prospective clinical trials: These approaches are not mutually exclusive. Clinician adjudication, for example, can provide the reference standard for retrospective testing, standardized interactions, or a prospective study. For interactive devices, however, retrospective evaluation and shadow deployments won’t work because the device’s response affects what the user says next. A previously collected interaction therefore cannot simply be replayed through a new model, and a silent model cannot participate naturally fit in the conversation. Limbic therefore did not use these approaches. Instead, they combining standardized patient interactions with independent clinician adjudication in a prospective, randomized study. The study enrolled 227 representative individuals who reported mental-well-being concerns such as low mood, low motivation, worry, or low self-confidence, but did not report being at risk of harming themselves. Participants were randomized to conduct a live, unscripted therapy-style session with one of three agent types: The transcripts were anonymized and evaluated by a consortium of 22 CBT-trained clinicians who were blinded to who—or what—had produced each session. They rated therapeutic competence using the Cognitive Therapy Rating Scale CTRS , an established rubric used to assess human CBT therapists. If blinded experts use this rubric to determine whether a person is delivering competent CBT, it can serve the same role for an AI. Limbic’s approach is consistent with FDA’s suggestion: “Where device outputs are evaluated against scoring rubrics, such rubrics should be grounded in established clinical guidelines or validated through qualified expert input and fit-for-use within the assessment context.” The results were striking. The agents wrapped in Limbic’s cognitive layer scored substantially higher than the standalone LLMs mean CTRS 4.5 versus 3.2 and also outperformed the human therapists. About three-quarters of the AI-led sessions exceeded a threshold reached by only the top tenth of human-led sessions. In blinded head-to-head comparisons, the expert panel preferred the layer-wrapped sessions on all seven criteria—including whether they would trust the agent with their own patients. Two caveats belong next to those results. First, CTRS measures the process quality visible in a therapy transcript, not whether patients ultimately improve. Second, the study evaluated single sessions with volunteers, not complete treatment episodes. In their Generative AI discussion paper, CDRH says they’re considering whether its appropriate to accept a greater level of pre-market uncertainty for GenAI-enabled devices in exchange for greater reliance on post market monitoring. The entire TEMPO program emphasizes this approach, as the point is to let devices get on the market sooner. In Limbic’s case, they were already on the market in the UK and had a publicly available well-being app in North America. They were therefore able to analyze substantial real-world usage data: 19,674 conversations from 8,920 users, including 8,435 users seeking well-being support through the publicly available North American app. Given the large volume of cases, they did not have clinical experts evaluate all of them. Instead, they developed an “LLM evaluator” that would apply the CTRS scores to each session. They validated this LLM evaluator against the transcripts that were scored in the prospective trial. In addition, 128 transcripts were also independently scored by the clinical experts. I think this will be a common pattern for generative AI-enabled devices my partner Yujan had predicted this even a couple years ago . The volume involved in post-market surveillance makes this approach necessary. Beyond benchmarking, clinical confirmation, and post-market monitoring, the study addresses another problem that every device built on a foundation model will face: the model underneath the device will change. Each time Open AI or Anthropic releases a new versions they also retire old ones. Limbic built this problem into its study design. The researchers tested four foundation models—GPT-4, Claude, Gemini, and Llama 3—both on their own and wrapped in Limbic’s cognitive layer. The bare models varied in quality, but the wrapped models converged. This result suggests that much of the clinical performance comes from the layer around the LLM, not from the LLM alone. It also provides a useful test of how much performance changes when the foundation model is swapped. FDA has raised this exact issue in their discussion paper: how can a manufacturer show that a change it did not initiate will not harm the device’s safety or effectiveness? Treating the foundation model as a variable in a pivotal study could help support a predetermined change control plan rather than a new submission for every model change. The finding also connects back to Unpacked’s architecture. Its performance is likely less sensitive to the conversational LLM because Limbic built substantial scaffolding around the model. That scaffolding adds control: the device behaves more predictably and is more robust to model swaps and updates. But the same controls create a tradeoff. As foundation models improve, the device may not gain their new abilities automatically because the surrounding layer still limits what the model can do. I believe this is a tradeoff most generative AI manufacturers will face. Limbic’s device matters because its LLM performs clinically meaningful work in a device now on the US market through TEMPO. Its Nature Medicine paper is therefore one of the clearest public examples we have of the evidence FDA may find acceptable for this new category of device. Read alongside FDA’s recent discussion paper, the study turns FDA’s abstract framework into a concrete example of how benchmarking, clinical confirmation, and post-market surveillance can work together. For interactive generative AI-enabled devices, I believe prospective studies will often be necessary, with sample sizes on the order of hundreds. Those studies will likely need to be supported by larger-scale benchmarking of individual system components and by post-market surveillance that uses LLM evaluators to make ongoing review feasible at scale. Limbic’s work provides an early example of how these complementary forms of evidence can be assembled. If you are developing a generative AI-enabled device, let’s get in touch Our team brings together engineering, clinical, and regulatory expertise to help clients develop evidence strategies and bring AI-enabled medical devices to market. We closely follow developments in this rapidly changing field so we can give clients practical, current advice, and we look forward to evaluating the evidence behind other generative AI-enabled devices as it becomes available. If you’re interested in the details of Limbic’s study, I suggest reading the full paper in Nature Medicine https://www.nature.com/articles/s41591-026-04278-w . Hi, I'm David Giese, a Partner at Innolitics. I'd love to connect on LinkedIn , where I post pragmatic tips about bringing AI-enabled medical software to market.