{"slug": "fleet-s-cto-tells-lds-how-it-keeps-ai-security-tests-isolated", "title": "Fleet's CTO tells LDS how it keeps AI security tests isolated", "summary": "Fleet Device Management is running Project Glasswing AI security tests inside temporary cloud deployments containing synthetic data, with customer endpoint data excluded, CTO Luke Heath told Lets Data Science. Fleet announced on September 15 that it joined Anthropic's defensive security program, and Heath said the work does not introduce a new Mythos-powered feature for customers. Access to the model is limited to named security team members using phishing-resistant multifactor authentication, and a security engineer must reproduce each candidate finding by hand before it is treated as real.", "body_md": "# Fleet's CTO tells LDS how it keeps AI security tests isolated\n\nIn written answers to LDS, Fleet CTO Luke Heath explains how Project Glasswing tests use isolated cloud deployments, synthetic data and engineer approval. Customer endpoint data is excluded, Fleet says. The interview separates those controls from the performance results still to come.\n\nFleet is testing what a frontier AI model can find in its software inside temporary cloud environments that contain synthetic data. In written answers to Lets Data Science, **Luke Heath, CTO of Fleet Device Management**, described where those tests run, what the model can do and why an engineer still has to reproduce a proposed finding.\n\nThe details give Fleet's participation in Project Glasswing a more concrete shape. The company announced on September 15 that it had joined Anthropic's defensive security program. Its work is focused on its own software and device-management protocols. Heath said the announcement **does not introduce a new Mythos-powered feature for customers**.\n\nFor teams assessing AI security tools, the useful question is what sits around the model: the data it receives, the authority it has and the evidence required before anyone acts on its output.\n\n### A temporary copy, with customer data left out\n\nHeath described three inputs to the research: Fleet's public source code; security context, including its threat model and previous advisories; and dedicated test environments. When the model needs to interact with a running system, the team deploys a temporary copy of Fleet in a separate cloud account containing synthetic data.\n\nHeath told LDS: \"**Customer endpoint data is not used in this work.**\"\n\nHe added that no data from customer environments is shared with the model or Anthropic. That is a statement about this research workflow. It should not be generalized into a claim about every AI integration a Fleet customer might separately use.\n\nWithin the test deployment, the model can execute test steps and probe the system. Heath said credentials are limited to that environment, a Fleet security engineer supervises the session, and anything extending beyond it requires that engineer's approval. The environment is removed afterward.\n\nAccess to the model is restricted to named members of Fleet's security team, individually authenticated with phishing-resistant multifactor authentication, according to Heath. He described these safeguards as already in place, while broader coverage across the release process remains planned.\n\nThis distinction matters when evaluating an agent. A convincing analysis is only one part of the system; the permissions around it determine what an incorrect action could affect. Fleet's account gives readers specific controls to ask about without establishing that LDS has independently audited their implementation.\n\n### A proposed finding has to survive human reproduction\n\nIn the exercise Heath described, the model approaches the temporary deployment from outside, without application credentials, and proposes possible attack paths. The human approval step determines which tests it may attempt.\n\n\"The approval step sits in the middle: a security engineer decides what the model may try, then **reproduces each candidate finding by hand** before anyone treats it as real.\"\n\nThat process separates a plausible model response from a vulnerability the team has actually verified. Heath said confirmed findings enter Fleet's existing remediation and disclosure process. Sometimes the model identifies a problem the team already knows about. Such a result can help check a capability, but it is not a newly discovered vulnerability.\n\nFleet's public [engineering handbook](https://fleetdm.com/handbook/engineering#handle-a-security-report) describes an engineer's initial review of security reports and a process for staging fixes while limiting disclosure of unresolved vulnerabilities. Its [security handbook](https://fleetdm.com/handbook/it/security#vulnerability-management) describes severity-based remediation and exceptions. These documents provide background on the process; they are not published results from the new Glasswing work.\n\nHeath did not disclose an individual Glasswing finding, a reproducible test result or measured performance improvement. He said detailed findings would follow after fixes have shipped and the material has been reviewed for safe disclosure.\n\n### Research access is different from a customer product\n\nAnthropic describes [Project Glasswing](https://www.anthropic.com/glasswing) as a defensive security initiative using Claude Mythos Preview. Fleet's own announcement and answers describe how it intends to apply that access to software and infrastructure it is authorized to test.\n\nHeath separately described endpoint-management functions Fleet already provides, including checking device state, deploying changes and reporting patch status. He said those existing functions do not depend on Glasswing or Mythos Preview.\n\nThat distinction keeps the announcement in proportion. Joining a research program does not establish that customers can use its model, that a new autonomous patching service has shipped, or that a measured improvement applies to their devices.\n\n### The result still to measure\n\nLDS asked how Fleet would determine whether AI improves on its existing security work. Heath said the team plans to compare AI-assisted work with its current engineering and security processes.\n\nThe measures he identified include valid issues missed by other tools or reviews, incorrect findings, human review effort, the accuracy of suggested fixes and the **time needed to reach a validated repair**. Fleet has not supplied results across enough Glasswing cases to support broad performance claims.\n\nFor a practitioner, these measures belong together. Finding more candidates is useful only if the team can establish which are real and resolve them. Faster model output can still leave engineers with more verification work. Fleet's answers set out what it plans to assess; they do not yet show the outcome of that assessment.\n\nThe patching survey cited around the announcement answers a separate question. Fleet's July report, [The road to AI in IT](https://fleetdm.com/pdfs/Fleet-IT-leadership-report_the-road-to-AI-in-IT.pdf), describes May fieldwork among senior IT leaders at organizations with at least 2,000 employees. It predates the September announcement and is not an evaluation of Glasswing.\n\nFleet supplied LDS with the exact patching question and its 527 responses. Of those respondents, **112, or 21.3% rounded**, selected less than 24 hours to deploy a critical security patch. These are respondents' estimates, not independently recorded installation times. The supplied question does not specify the timing boundaries, and two longer-duration answer labels overlap. Those limitations make it unsuitable as evidence that the new research has already improved patching.\n\n### Recovery must be checked on the device\n\nOne of Heath's practical points applies to ordinary automation as well as AI. A successful command does not establish that the intended change reached a device.\n\nHe explained that Fleet can query endpoint state after deployment to check the resulting software version, configuration or other relevant condition. Reverting the desired configuration after a failed change also needs a subsequent check that the device has returned to the expected state.\n\nHeath recommended that teams without access to Mythos Preview start by putting device policies and configurations into version control, requiring human review and testing changes on a small group before a wider rollout.\n\nThe same discipline can guide an evaluation of an AI tool: define what it may access, give proposed findings an independent check, and decide in advance what evidence will establish success or recovery. Fleet has provided a concrete account of those boundaries. The next reporting milestone is evidence of what its research finds and how much work it takes to turn a finding into a verified fix.\n\n### Reporting note\n\nBased on written answers from Luke Heath supplied directly to Lets Data Science on September 16, Fleet's September 15 announcement, its July survey report and public Fleet and Anthropic documentation. LDS checked the survey calculation and quoted answers but did not run the security tests or audit the controls. Descriptions of the research are attributed to Fleet. Portrait of Luke Heath supplied by Fleet for this article; used with his written permission.\n\n## Key Points\n\n- 1Fleet says its Glasswing research uses temporary isolated deployments with synthetic data and excludes customer endpoint data.\n- 2An engineer supervises the tests and reproduces candidate findings before they enter the remediation process.\n- 3No Fleet-specific Glasswing benchmark or new Mythos-powered customer feature has been established by these answers.\n\n## Scoring Rationale\n\nOriginal written answers to LDS from Fleet CTO Luke Heath explain isolated AI security tests, synthetic data, engineer approval, finding reproduction and endpoint verification. The reporting distinguishes operational controls from results still to be measured and from existing customer capabilities.\n\n## Sources\n\nOriginal reporting, with the public references used alongside it.\n\nLDS Exclusive\n\nReporting based on written answers given directly to Let's Data Science by **Luke Heath, CTO, Fleet Device Management**.\n\n## View 1 more source\n\nPractice interview problems based on real data\n\n1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.\n\n[Try 250 free problems](https://letsdatascience.com/problems)", "url": "https://wpnews.pro/news/fleet-s-cto-tells-lds-how-it-keeps-ai-security-tests-isolated", "canonical_source": "https://letsdatascience.com/news/fleets-cto-tells-lds-how-it-keeps-ai-security-tests-isolated-cffa99d7", "published_at": "2026-09-16 15:06:32+00:00", "updated_at": "2026-09-16 15:10:50.620716+00:00", "lang": "en", "topics": ["ai-safety", "ai-policy", "ai-agents", "artificial-intelligence"], "entities": ["Fleet Device Management", "Luke Heath", "Project Glasswing", "Anthropic", "Mythos", "Lets Data Science"], "alternates": {"html": "https://wpnews.pro/news/fleet-s-cto-tells-lds-how-it-keeps-ai-security-tests-isolated", "markdown": "https://wpnews.pro/news/fleet-s-cto-tells-lds-how-it-keeps-ai-security-tests-isolated.md", "text": "https://wpnews.pro/news/fleet-s-cto-tells-lds-how-it-keeps-ai-security-tests-isolated.txt", "jsonld": "https://wpnews.pro/news/fleet-s-cto-tells-lds-how-it-keeps-ai-security-tests-isolated.jsonld"}}