Safety Without Compromising on Privacy

Tinfoil is rolling out automated safety safeguards inside its Tinfoil Chat secure enclaves over the next few weeks, running open-weight models in hardware enclaves so conversations stay invisible to everyone but the end user. Tinfoil will not host any model that responds inappropriately to more than five percent of questions mapping to its three harmful categories: encouraging depression and self-harm, mass violence and terrorism, and child abuse. The company says running safeguards inside enclaves means it cannot tune its pipeline on customer data or human-review flagged conversations, but it will publish a Safety & Safeguards page so anyone can audit how the policy is enforced.

← Back to Posts https://tinfoil.sh/blog Safety Without Compromising on Privacy Tinfoil runs open-weight models inside hardware enclaves so that conversations are invisible to everyone aside from the end user. As models become more powerful, their potential for harm also increases. By providing access to the most powerful open-weight AI models in a way that is fully private, we're opening the doors to important safety concerns. This is especially true as we look to expand the capabilities of our chat by supporting video and image generation models. We cannot allow a few users to abuse the privacy offered by Tinfoil to generate child sexual abuse material or get real help planning mass violence. Nor can we allow our models to fail to effectively help a person in crisis. Companies like OpenAI and Anthropic deal with safety concerns by running automated classifiers and models on all conversations, progressively escalating to async analysis and human review.<sup 1 user-content-fn-1 </sup We believe we can get most of the safety benefits with fully automated safeguards that run exclusively inside secure enclaves and don't expose conversations. There are some disadvantages to running safeguards inside secure enclaves. For example, we cannot tune our safeguards pipeline on our customers' data or perform human review of flagged conversations to resolve false positives. However, we think these limitations are less of a problem than they might seem when considering open-source and open-weight models. Open source provides a path forward We are big proponents of open source all our security-critical code is open source , and it helps us in two ways here. 1. All our hosted models are open-weight and widely available via other providers. This means that the models can be tested and evaluated by many independent parties to surface safety concerns. We think there is a lot that can be done in this space. 2. Open-source datasets and safeguard models are available to us. The downside of open-source safeguards and datasets is that they can be trained against and bypassed. However, as models get bigger they generalize better, making it more difficult to deceive them. We believe that we can get realistic enough data to be confident about how our safeguards will perform in production before deploying them. Furthermore, the Tinfoil approach to safeguards comes with its own advantages. For example, we can prove to an auditor, a regulator, or anyone else who asks, that the safeguards are running, and prove exactly what the safeguard policy is. We can do so because the entire pipeline code is public and its enforcement is verifiable through attestation. By building in the open, we hope to collaborate and build on safeguard work done by institutions that know more about safety than we do, instead of rolling our own. We're excited to see the ecosystem of open safeguards and data grow, and hope to make our own contributions. To this end, we are rolling out safeguards in Tinfoil Chat over the next few weeks, together with a Safety & Safeguards page that lays out the policy and lets anyone audit how it is enforced. Model evaluations and chat safeguards We implement a two-layered approach: First, before we host a model, we test it against our policy. The policy lists three harmful categories: 1 encouraging depression and self-harm, 2 mass violence and terrorism, and 3 child abuse. A model that responds inappropriately to more than five percent of the questions that map onto these categories is not hosted. We make these categories specific and minimal to avoid infringing upon the personal liberty of our users. Second, in our private chat, we run safeguard models at inference time. These safeguard models run inside secure enclaves with the same privacy and security guarantees as all our other models and flag harmful conversations that violate the policy above. If they find a violation, the only thing that leaves the enclave is that a violation occurred, attached to your account and the ID of the flagged conversation. The conversation itself stays private, as does the category that was violated. This information is used to notify the user upon a flag, and repeated violations will lead to the account being banned. Is it possible to have both privacy and safety? Different AI companies have their own set of values around how they deal with this tension. Anthropic, for instance, rolled back zero data retention ZDR for their Fable and Mythos models https://support.claude.com/en/articles/15425996-data-retention-practices-for-covered-models and required 30-day data retention for safety purposes. However, following OpenAI, they recently carved out some exceptions to allow a handful of large enterprises to run safeguards on their own cloud infrastructure and have human review be performed by their own employees. We think our solution of running safeguards inside secure enclaves allows everyone to benefit from strong privacy, not just a select few. In the past, proposals for content safety monitoring of end-to-end encrypted messaging have required weakening the security guarantees of the system by introducing backdoors. At Tinfoil, everything runs inside secure enclaves. This makes our infrastructure capable of implementing certain safeguards without changing the privacy guarantees of the system. Privacy absolutists will say that once any type of scanner exists in a privacy-preserving system, the only outcome is mass surveillance. However, we believe that for the long-term preservation of individual freedom and privacy, there needs to be a responsible policy for responding to abusers that may ruin the promise for everyone, a la, this is why we can't have nice things . We also strongly believe that it is important to demonstrate publicly that it is possible for both privacy and safety to coexist, such that arguments that claim that the only people who want privacy are bad actors are not perpetuated. The trending narrative is that in order to provide safety, one has to fully sacrifice all individual privacy and freedom. In functioning democracies, the constitution and the laws themselves are public. As with everything else in the Tinfoil infrastructure, we lay out the specific safeguards we use as well as the enforcement mechanism in a publicly verifiable, auditable, and transparent manner. To this end, our evaluation benchmarks and safeguards are all open source https://github.com/tinfoilsh/safeguard-evals , so that anyone can audit what is going on or why a conversation was flagged. Of course, by making policies transparent, we run the risk of malicious entities optimizing around them. We think that this is not a concern for our specific use case here, but believe this is an important consideration and something deserving of more thought. The hard-nos of our acceptable use policy We develop the following simple and well-defined hard-no policy outlining what constitutes a violation, and try to make it as clear and objective as possible. There are three categories to our hard-no policy which we use to evaluate models and monitor chat conversations: - Encouraging depression and self-harm : Existing lawsuits https://www.cnn.com/2025/11/06/us/openai-chatgpt-suicide-lawsuit-invs-vis against OpenAI are mostly focused on their 4o models and sycophancy. In many of these, the models encouraged the user to see the world in warped ways, and played into their depression, isolation, and paranoia. Our hosted models should not encourage or worsen suicidal ideation or action. - Mass violence and terrorism : Models can uplift and make it easier to create or access CBRN weapons and can be used to help plan and commit mass acts of violence or terrorism. Our hosted models should not generate content providing real-world uplift to mass violence or terrorism, including production of CBRN weapons. - Child endangerment : There are many real-world ways models can facilitate child endangerment, for instance encouraging users to take action that is harmful to children. Our hosted models should not do this, and also must not generate explicit sexual content depicting minors. The exact hard-no policy can be seen verbatim on our Safety & Safeguards https://tinfoil.sh/safety-and-safeguards safeguards page. It remains the user's responsibility to comply with our full terms and conditions https://tinfoil.sh/terms , which are much broader, and contain a superset of things that constitute unacceptable use. For example, our acceptable use policy prohibits generating sexually explicit content involving real, identifiable people without their consent, and using Tinfoil to facilitate the distribution of spam or malware. Our terms of service and hard-no policy are both subject to change, especially as model capabilities improve. Technical details and benchmarks Evaluating models before we host them Most open-weight models come with some degree of safeguards built in, which is done by training models to respond appropriately and refuse to provide harmful information. However, while the weights of the models we host are open, the data they are trained on is not public. Some models come with model cards explaining what they were evaluated on, but many, especially open-weight models, do not have this information publicly available. For example, in Gemma's Google's open-weight model system card https://ai.google.dev/gemma/docs/core/model card 4 , the Ethics and Safety section provides minimal insight into how the evaluations were performed despite listing that Google aims to "prevent generation of harmful content." The closest thing to a model card https://z.ai/blog/glm-5.2 for GLM 5.2 has no mention of safety evals whatsoever. To judge whether a model is acceptable to host on Tinfoil, we evaluate how each model performs in its responses on a curated dataset of prompts from a selection of open-source safety questions. More specifically, we take 1,500 prompts from the MLCommons AILuminate sample dataset https://github.com/mlcommons/ailuminate and from the HarmBench dataset https://github.com/centerforaisafety/HarmBench .<sup 2 user-content-fn-3 </sup A first pass with an LLM-as-a-judge flagged 406 prompts as candidate matches for our hard-no policy. We then reviewed the full set by hand, keeping 281 of the flagged prompts and adding 16 the judge missed, settling on 297 prompts that unambiguously fall into one of our three hard-no categories a model should not engage in. Release decisions are made only based on how the model performs on this 297-prompt set. If a model gets more than 5% of the hard-no questions wrong, we will not serve it on Tinfoil. Note that even when models do well on these benchmarks they may perform poorly in real use: these are just some prompts, out of a massive amount of potential conversations. Figure 1 shows results for the models we serve today. You can read our criteria and more information on our evals, and how every text model we serve performs on them, on our website https://tinfoil.sh/safety-and-safeguards evals . You can also see our evaluation code, data, and granular results for each model in our open-source safeguard-evals repository https://github.com/tinfoilsh/safeguard-evals . We include how the models perform on all questions, not just our hard-no subset, so that users can make informed decisions. For example, DeepSeek V4 Flash failed to respond correctly to a prompt asking it to manufacture the smallpox virus: Prompt Response Active safeguards in chat in-enclave content review Our evals can only say so much: we test our models on a small number of conversations, and models are non-deterministic, even at temperature 0. Minuscule changes to one token can cascade. This means that models could perform differently during actual use. Additionally, if a model gets some hard-no questions wrong, but not more than 5%, we want to continue serving it, but we don't want it to answer those in production. Finally, jailbreaking, or adversarial prompting, can be used to get around built-in safety training. For these reasons, to make sure our hard-no policy is enforced in our chat application, we have additional safeguard models that are designed to evaluate conversations during runtime to flag harmful responses from the model. The safeguard pipeline runs inside secure enclaves and only outputs a flag if it determines a conversation violates our hard-no policy. The safeguard pipeline is designed not to reveal anything about the conversation or the nature of the violation. The exact policy the safeguard models judge by can be seen on our Safety & Safeguards page https://tinfoil.sh/safety-and-safeguards safeguards or in the confidential-safeguard repository https://github.com/tinfoilsh/confidential-safeguards . Note that it is a fine-grained specification of our hard-no categories listed above, but should be semantically the same. At the time of writing it contains five categories, as child abuse and mass harm have been split into two categories apiece for ease of judging. In practice, what constitutes a jailbreak can be ill-defined. For instance, using models to write a fictional character is a common use case, but many jailbreaks also employ fictional framing to try and get around safety training. Genuine fictional content that doesn't help the user in achieving dangerous real-world outcomes is fine, while a fictional narrative designed to assist the user in performing harm is not. Instead of looking for jailbreaks in user prompts, our safeguards check whether the model's response in the context of the conversation violates our hard-no policy. Tuning the pipeline to minimize false positives Since we can't human-review any conversation that happens in Tinfoil even if we wanted to , we need to make sure our safeguard models get the most complete context they can and have a near-zero false-positive rate. To simulate our pipeline on production chat traffic containing harmful content, we obtained access to the WildChat-4.8M-Full dataset https://huggingface.co/datasets/allenai/WildChat-4.8M-Full , a public corpus of real ChatGPT conversations.<sup 3 user-content-fn-2 </sup The total dataset contains ~4.8 million real conversations. We pre-processed this dataset to remove bot conversations by clustering conversations with similar prefixes, and then manually identifying different pathologies, that is, repeated structured prompts. After this we were left with 1.2M conversations, with ~266,000 labeled as toxic by WildChat. For our first pass we use a small model, specifically gpt-oss-safeguard 120B , which is trained to follow a precise written policy. To minimize false positives, we have an additional step in the safeguard pipeline where we use an even smarter model, Kimi-K3, to assess the conversation. To validate our policy was being followed correctly, we ran this process on all the WildChat conversations, and then manually reviewed performance, as shown in Figures 2 and 3. We tested multiple models for our first-pass model. On a 50k subset we tried running gpt-oss-safeguard and gpt-oss, with the same prompt. Our prior was that oss-safeguard would over-flag due to its safety training not being the same as our prompt, but actually it was the opposite: safeguard followed the specifications in the prompt better than gpt-oss. Figures 4 and 5 show the comparison. Based on these results we use gpt-oss-safeguard. We show the safeguard model the full conversational context to avoid small safety classifiers getting confused and triggering false positives. We also explored smaller per-turn classifiers, as a first pass to narrow down, but decided against them because: - They tend to be sensitive to how the prompt is phrased. - Since they can't see the full conversation they can under-flag, making them unusable as a first pass. - The latency improvement isn't strictly necessary here since we process out of band. To further reduce the possibility of false positives from gpt-oss, our safeguards pipeline has a second opinion from a large model, Kimi-K3 in testing, that decides if a chat is indeed in violation of the hard-no policy. Prompt Response Evaluating the model response Our safeguards run in parallel to the inference request; they do not block a response or add latency to the chat responses. Because it can take time for the moderation enclave to drain, the safeguard enclave could change versions before reviewing a conversation. When a version changes all to be reviewed conversation data is lost. No conversation data is persisted between versions. As with everything else hosted on Tinfoil, you don't have to take our word for it. Instead, you can go see how everything works for yourself. All our code is open source and you can view the router enclave https://github.com/tinfoilsh/confidential-model-router and the confidential safeguard https://github.com/tinfoilsh/confidential-safeguards on our GitHub. Moreover, our infrastructure visibility page https://visibility.tinfoil.sh/ makes auditing our code easier and shows how everything fits together. What happens when a conversation is flagged If the safeguard flags a conversation, you get a notice in the chat app and via email. The only record Tinfoil holds is the flagged conversation's ID, as this is necessary for us to notify you. If your account accumulates a large number of flags over a period of time, we automatically suspend it. Our safeguards are set up to be safe in realistic ways, and focus on preventing actual harm. For example, the OpenAI lawsuits show increasing harmful usage across time and conversations, as users entered increasing levels of AI psychosis. This is why we choose to ban users only after a large number of repeated violations. We do not want to police how our models are used or ban users for incorrect classifications. If your account is banned, you can email us to discuss reinstatement. Because we cannot see the conversations that were flagged, reinstatement of the account will require you to prove to us that your usage didn't violate our acceptable use policy. This could be done in one of two ways: either 1 by sharing your flagged conversations with us out-of-band if you believe our safeguards classified it incorrectly, or 2 by proving to us that your use of Tinfoil doesn't violate our acceptable use policy e.g., you're a safety researcher testing our models . What we don't do Our approach to safeguards stands in contrast to many prior industry proposals for content safety monitoring. In particular, proposals for monitoring encrypted messaging applications often require weakening the trust model and privacy guarantees by planting backdoors into end-to-end encrypted systems or running classifiers on a user's device without clear scanning boundaries. An example of this is the canceled proposal https://www.wired.com/story/apple-csam-scanning-heat-initiative-letter/ by Apple to scan cloud storage for CSAM. The proposal involved scanning end-to-end encrypted conversations stored in iCloud. Methods of doing this would weaken the privacy provided by end-to-end encryption. To quote Apple's letter to the HEAT Initiative https://heatinitiative.org/ around why they canceled this, the main issues with their proposal were that "scanning for one type of content ... opens the door for bulk surveillance" and there was no way for "users to be assured that a tool for one type of surveillance has not been reconfigured to surveil for other content." Our approach to safeguards does not weaken any existing privacy guarantees around conversation data: all backed-up and synced chats are stored fully end-to-end encrypted on our infrastructure and are not subjected to any safeguards . The safeguards we will be rolling out exclusively run inside secure enclaves and only scan responses at inference time, making everything ephemeral and confined to encrypted memory. We are able to prove to users exactly what content is scanned for because all our code that touches any conversation is fully open source and attested. Closing thoughts We believe privacy and safeguards can coexist. We want to lead the way in doing so by building in an open and verifiable way. We've worked to align our policies with real-world harm by looking through things like the OpenAI lawsuits https://techjusticelaw.org/announcements/seven-ai-delusional-disorder-cases-shamblin-irwin-fox-enneking-madden-brooks-and-lacey-v-openai-et-al/ , open-source conversations from real users, and existing benchmarks and evals. As always, we welcome feedback on how to improve our product. At Tinfoil we want to cultivate the space to talk about anything, in private, with the most powerful intelligence available. We want to continue to exist as a company and provide a necessary and safe service that can be used by everyone. This is one step we're taking towards making the private garden accessible to those who care, while keeping away those who choose to abuse it. Acknowledgements We'd like to thank Addie Foote, Alejandro Tlaie Boria, Anthony Ronning, Kurt Opsal, and our anonymous chat users for helpful comments on earlier drafts of this post. Footnotes 1. See data usage and retention policies from OpenAI https://help.openai.com/en/articles/7039943-data-usage-for-consumer-services-faq and Anthropic https://support.claude.com/en/articles/8325621-i-would-like-to-input-sensitive-data-into-my-chats-with-claude-who-can-view-my-conversations?q=thumbs , and the OpenAI https://openai.com/trust-and-transparency/ and Anthropic https://www.anthropic.com/transparency/system-trust-reporting transparency reports. Looking through the OpenAI government requests for data, we can see that the number of requests for content, and number of emergency requests, is increasing. In 2023 it was 0 for content and for emergency requests. In 2025 it was 75 and 10 requests, respectively. ↩ user-content-fnref-1 2. The AILuminate v1.0 DEMO Prompt Set is under the CC BY 4.0 license https://creativecommons.org/licenses/by/4.0/legalcode . HarmBench is under the MIT License https://github.com/centerforaisafety/HarmBench/blob/main/LICENSE . ↩ user-content-fnref-3 3. Our preprocessing code and prefixes can be found in our safeguard-evals repository https://github.com/tinfoilsh/safeguard-evals . WildChat-4.8M-Full is under the ODC Attribution 1.0 license https://huggingface.co/datasets/allenai/WildChat/raw/main/LICENSE.md . ↩ user-content-fnref-2 Subscribe for Updates RSS Feed https://tinfoil.sh/feed.xml Stay up to date with our latest blog posts and announcements.