{"slug": "visionos-27-is-a-much-bigger-deal-than-apple-s-keynote-suggested", "title": "visionOS 27 Is A Much Bigger Deal Than Apple's Keynote Suggested", "summary": "Apple announced visionOS 27 at WWDC 2026, the fourth major OS update for Apple Vision Pro, with features including held object tracking, IR LED tracked accessories, unfoveated screen recording, and a Siri AI overhaul powered by Apple Foundation Models co-developed with Google. The update, arriving this fall for M2 and M5 models, also introduces spatial panorama environments and a dedicated Siri app, marking one of the biggest upgrades yet despite minimal keynote coverage.", "body_md": "There's a whole lot more coming in visionOS 27 than what Apple announced in its keynote and blog post.\n\nAnnounced by Apple at WWDC 2026 this week, visionOS 27 will be the fourth major operating system version for Apple Vision Pro headsets.\n\nWhile Apple's main WWDC 2026 barely mentioned the Vision Pro operating system, its documentation and developer talks reveal that visionOS 27 is one of the biggest upgrades yet, with (preconfigured) held object tracking, an IR LED tracked accessories framework, unfoveated screen recording, and a range of other major improvements.\n\nvisionOS 27 will arrive \"this fall\", for both the M2 and M5 models of Apple Vision Pro. The first beta release of visionOS 27 is available now, though it lacks many of the features coming in the stable launch.\n\n# visionOS 27\n\n## Siri AI & Visual Intelligence\n\nThough not coming with the initial visionOS 27 release, Apple says that \"later this year\" Siri AI, an LLM-powered overhaul of its digital assistant, will come to its major operating systems, including visionOS.\n\nSiri AI is powered by the next-generation Apple Foundation Models, which were co-developed with Google and based on its Gemini models ([reportedly](https://www.bloomberg.com/news/articles/2025-11-05/apple-plans-to-use-1-2-trillion-parameter-google-gemini-model-to-power-new-siri?ref=uploadvr.com) via a $1 billion/year deal). As has always been the case with Apple Intelligence, some tasks run on a small on-device model, while more complex tasks are handed off to Apple's Private Cloud Compute system.\n\nApple says Siri AI is a \"profoundly more capable and conversation assistant\" than traditional Siri, with \"personal context understanding, broad world knowledge, and more\".\n\nIn visionOS, Siri AI will take the form of a persistent orb that you can pin to a physical spot in your room. You can start speaking to Siri AI by simply looking at the orb and speaking, bypassing the need to say \"Siri\", though you can still do that if you want.\n\nSiri AI on visionOS will also support Apple's Visual Intelligence feature, letting you \"search, ask questions, and take action\" relating to your physical and virtual surroundings. In terms of capabilities, it seems to sit somewhere between Meta AI on Quest 3's Horizon OS and Gemini on Samsung Galaxy XR's Android XR.\n\nvisionOS 27 also includes a dedicated Siri app, letting you access previous conversations, and offering a preinstalled alternative to third-party AI chatbot apps like ChatGPT, Gemini, Claude, Grok, and Meta AI.\n\nThe next-generation Apple Foundation Models will also bring:\n\n- Significantly more realistic image generation in Image Playground.\n- Automatic tab organization in Safari.\n- The ability to upgrade the strength of your saved passwords with a single tap.\n- Personalized Smart Replies in Messages.\n\n## Spatial Panorama Environments\n\nvisionOS 2 brought the ability to add synthetic stereoscopy to 2D images, and then visionOS 26 brought the ability to turn 2D images into volumetric spatial scenes, with a deep sense of perspective enabled by Gaussian splatting.\n\nvisionOS 27 extends this spatial scenes concept to your iPhone-captured panoramas.\n\nYou can set these spatial panoramas as your visionOS home environment, meaning they can be the background for your windows and volumes in the Shared Space.\n\n**Higher Frequency Object Tracking**\n\nBack in visionOS 2, Apple [added](https://www.uploadvr.com/visionos-2-3d-object-tracking/) the ability for Vision Pro to track specific real-world 3D objects that app developers prespecify.\n\nDevelopers provide a 3D model of the object they want to be tracked to the Create ML tool on their Mac before app compilation, and over the course of several hours it trains a neural network for the object and bakes it into the app.\n\nWhile genuinely useful for enterprise applications, the visionOS 2 object tracking system was only designed to be used for static objects, mainly in order to anchor virtual content to them, and updated at around 5Hz (though in practice often slower).\n\nvisionOS 27 adds a high-frame rate object tracking mode that is suitable for moving and held objects. While Apple doesn't say exactly what the update rate is, initial reports from developers suggest it may be around 30Hz.\n\nTo track held objects, Apple strongly recommends using the new Extended Training mode, which greatly improves tracking performance though also \"significantly\" increases the machine learning model training time. XR enthusiast Brad Lynch found that it took around 18 hours to [train a fish toy](https://x.com/SadlyItsBradley/status/2065974984258818512?ref=uploadvr.com), for example.\n\nWhile high frame rate object tracking is leagues better than what came before, it's important to note that it still has a significantly lower update rate, and thus higher latency, than hand tracking or tracked accessories. And speaking of tracked accessories, we're likely to see a lot more of them soon.\n\n## IR LED Tracked Accessories\n\nFor visionOS 26 last year, Apple worked closely with Sony and Logitech to add hardcoded support for the PlayStation VR2 Sense controllers and Muse digital stylus, which both have a constellation of infrared LEDs under the plastic of their body.\n\nWith visionOS 27, Apple has opened this process up to any interested company by adding the necessary APIs to enable them to make their own IR LED tracked accessories for Vision Pro, similar to how any company can make SteamVR \"Lighthouse\" Tracking accessories on PC.\n\nOptical tracking runs at 90Hz, and this is fused with the IMU accelerometer and gyroscope data from the object, enabling the same quality of tracking you'd expect from PlayStation VR2 Sense controllers.\n\nThere are two possible software support paths available.\n\n**Companion App:** The hardware maker can release a visionOS companion app for the hardware, essentially an installer for it, that registers the shape of the object and configuration of its IR LEDs with visionOS so that other apps can use them. This path would be highly suitable for a consumer device.**Built-In To App:** Alternatively, a developer can build the object details into an app that utilizes the accessory, negating the need for a separate device companion app. This path would be ideal for enterprise.\n\nDFRobot and MikroE are making reference hardware and developer kits for this, and will release them later this year. Companies can use these directly, or as the basis of their own accessories, lowering the barrier to entry compared to building from scratch.\n\n## Thórsmörk Environment\n\nvisionOS 27 will bring a new environment option for the Shared Space: Iceland's Thórsmörk valley. It translates as Thor's Valley, and it's widely considered one of the most scenic places on the planet.\n\nApple's environments all have day and night modes, and Thórsmörk's nighttime variant features a dramatic aurora borealis.\n\nThórsmörk will be the 10th virtual environment for visionOS, following the addition of [Jupiter from Amalthea](https://www.uploadvr.com/apple-visionos-jupiter-environment-shadow/) in visionOS 26.\n\n#### All visionOS Environments\n\n— summit of the Hawaiian volcano**Haleakalā**— wintry Yosemite National Park** Yosemite**— desert landscape in Joshua Tree National Park** Joshua Tree**— Trillium Lake facing Mount Hood** Mount Hood**— White Sands National Park** White Sands**— the lunar surface** The Moon**— lakeside environment in Norway; added to visionOS 1 in July 2024** Lake Vrangla**— French Polynesian beach; added with visionOS 2** Bora Bora**— viewed from Amalthea, one of Jupiter’s inner moons; added with visionOS 26** Jupiter**— Icelandic highlands, with an icy lake, mountains, snow and Northern Lights; announced for visionOS 27** Thórsmörk**\n\n## Unfoveated Screen Recording\n\nAt all times in the Shared Space, and in many Full Space apps too, Apple Vision Pro uses always-on eye-tracked foveated rendering, only rendering the region of the image you're currently looking at in full resolution.\n\nThis maximizes visual quality and performance, and is essential to how Apple Vision Pro headsets make good use of their 4K micro-OLED panels, but it has the tradeoff of meaning that in screen recordings, everywhere you're not looking is a low-resolution blur.\n\nIt was possible to record unfoveated footage in the visionOS Simulator on a powerful enough Mac, but not on the headset.\n\nvisionOS 27 adds a High Quality Recording option, which records up to 3 minutes of unfoveated 4K footage, completely on the headset, no Mac required.\n\nIt should make it much easier for developers, journalists, influencers, and content creators of all kinds to show off visionOS, and was one of the most highly requested features for Vision Pro.\n\n## Mac 3D Object Preview API\n\nWhen connected to your Mac via Mac Virtual Display, macOS 27 adds an API to spawn a 3D object on visionOS 27, letting you preview objects in 3D space.\n\nApple's built-in Preview macOS app supports it, meaning you can open a 3D object from Finder and preview it in seconds. And developers can add it to their own Mac apps too.\n\nThis API essentially lets developers add Vision Pro support to their 3D creation Mac apps with just a few lines of code, and without the need to build a visionOS app.\n\nIt even supports editing the model on the visionOS side using gaze-and-pinch controls, and these edits will then be reflected on the macOS side. It's a two-way API. SharePlay is supported too, for collaborative spatial editing with Personas.\n\nApple says Cinema 4D and SketchUp are working on integrating the API, and we expect to see many more Mac apps do so once macOS 27 and visionOS 27 ship.\n\nMicrosoft and Meta announced something similar for Windows 11 and Horizon OS [back in 2024](https://www.uploadvr.com/microsoft-windows-volumetric-apps-quest/), but as far as we're aware, it has yet to ship.\n\n## New Control Center Look\n\nvisionOS 27 updates the look of Control Center, moving it into three panels for a larger and less crowded total space for the interface.\n\n- The left panel shows the date, time, battery level, Wi-Fi status, and playing notifications.\n- The center panel shows shortcuts to Wi-Fi, Bluetooth, AirPlay and Airplane Mode, as well as a volume slider and quick controls like Mac Virtual Display, Mirroring, Guest Mode, and Travel Mode.\n- The right panel controls the environment, including its day/night mode and the spectrum between full passthrough and full enclosure in the virtual environment, as you would with the digital crown physical control.\n\n## Mac Device Hub\n\nXcode 27 for macOS brings a Device Hub app for developers.\n\nFor developers, Device Hub should make running Apple Vision Pro demos much smoother, as instead of needing to guide the guest through the visionOS gaze-and-pinch interface, they can instead just remotely launch an experience.\n\nIt's a significant addition because the iPhone and iPad companion apps for Vision Pro cannot launch apps, for some reason.\n\n## 3x Faster Wi-Fi Initialization\n\nThe M5 Apple Vision Pro is a very fast-feeling device, and in my several months with it I've experienced no slowdowns.\n\nThere is one strange exception, though. After booting up the device, for the first 20 seconds or so there is no internet connection. It takes an unusually long time, compared to all of the other devices I own, to fully connect to Wi-Fi.\n\nIt seems that Apple is not unaware of this, as in visionOS 27, the company claims Vision Pro headsets will connect to Wi-Fi up to three times faster.\n\n## Curved Windows In Safari, Freeform & Multiview\n\nBefore visionOS 27, all windows were flat, with the exception of Mac Virtual Display's ultrawide mode which is curved.\n\nvisionOS 27 adds curved windows support to Safari, Freeform, and Multiview on Apple TV. Safari also makes use of this for a curved tab browsing experience that somewhat wraps around you, as does the Multiview mode of Apple TV.\n\nUnfortunately, this is a private API for first-party Apple apps. Third party developers cannot make their windows curved – at least not in the Shared Space. In a Full Space, developers can implement a custom rendering solution for this, but this will come with significant tradeoffs in terms of API support and OS consistency.\n\n## Extra-Small 'Accessory Widgets'\n\nOne of the biggest additions in [visionOS 26](https://www.uploadvr.com/visionos-26-announced-apple-vision-pro-wwdc25/) was Widgets, bringing persistent mini-windows you can pin to your physical space, including Clock, Weather, Music, and Photos, alongside an API for developers to build their own.\n\nvisionOS 27 adds a new kind of extra small widget called accessory widgets. They're smaller than visionOS 26's widgets and could be used for things like a battery level, countdown or timer, a stock price, smart home control button, or shortcut into an app.\n\nThe smart home button possibility seems particularly promising. You could, in theory given an app that supports this, add accessory widgets around your home on your physical devices, letting you just look at them and pinch to activate or deactivate them.\n\nApple has added a first-party accessory widget for Mac Virtual Display, which launches a Mac Virtual Display session for a given Mac, even when it's locked. While visionOS already creates a button above your MacBook, this could be particularly useful for Mac mini and Mac Studio, letting you just sit down at your desk and tap a virtual button to connect.\n\n## Web Environments In Safari\n\nSafari in visionOS 27 adds support for Web Environments, letting developers specify a 3D environment for their webpage that users can optionally enable to surround them.\n\nThis is separate from WebXR, more akin to how web developers set a favicon than anything involving active JavaScript or anything like that.\n\n## Gaze-Expanding Notifications\n\nIn visionOS 27, notifications can expand when you simply look at them, exposing more information and available actions without the need to pinch.\n\nIt's another example of using eye tracking to reduce user interface friction.\n\n**Auto Subtitles For Videos**\n\nvisionOS 27 adds an Accessibility feature to automatically generate subtitles for any video, using an on-device speech recognition model.\n\n**Vehicle Motion Cues**\n\nvisionOS 27 is bringing the Vehicle Motion Cues feature that Apple added to its other major operating systems in 2024.\n\nOn Apple's other devices, motion cues appear as animated dots around the edge of the display that move in response to the acceleration, braking, and turning of the vehicle you're in, with the intention of reducing travel sickness.\n\nIt's unclear how Vehicle Motion Cues will work in visionOS, and whether they might have applications for reducing motion sickness in VR apps too.\n\n**Eye-Controlled Power Wheelchairs**\n\nvisionOS 27 adds the ability to natively control select power wheelchairs using eye tracking.\n\nApple says the feature will launch with Tolt and LUCI alternative drive systems in the US, with support for both Bluetooth and wired connections.\n\nThe company says it \"will continue to work with developers to expand support for more wheelchair drive systems\".\n\n# Major RealityKit Upgrades\n\nvisionOS 27 also brings major upgrades to RealityKit, Apple's built-in \"engine\" that powers all experiences in the Shared Space and many Full Space apps too.\n\nRealityKit allows developers to build high-quality VR and mixed reality experiences on Vision Pro without a heavy engine like Unity or Unreal, with consistent rendering, physics, audio, and interactions.\n\nHere are the key upgrades coming to RealityKit in visionOS 27:\n\n### Physical Space Lighting\n\nApps can now cast virtual light on your physical environment.\n\nRealityKit already automatically has virtual objects and interfaces cast shadows into your physical space, based on where it estimates your real lighting is coming from, and this has been a key feature since visionOS. What's new in visionOS 27 is the ability to cast light too.\n\nUnder the hood, the feature leverages Vision Pro's LiDAR-scanned 3D mesh of your room, alongside the reading from the ambient light sensor. The rest is clever software.\n\nDevelopers can add both point and spot lights to the scene, with support for projective textures to add a pattern (optionally animated) to spot lights for effects such as stained glass, underwater caustics, projected stars, or patterned stage lighting.\n\n### Native Gaussian Splat Rendering\n\nOver the past few years, to use an analogy, Gaussian splatting – fitting millions of semitransparent colored blobs (Gaussians) in 3D space so that arbitrary viewpoints can be rendered realistically in real-time – has done for 3D reconstruction what [transformers](https://en.wikipedia.org/wiki/Transformer_(deep_learning)?ref=uploadvr.com) did for AI.\n\nWe've seen developers do amazing things with it, such as Gracia's streamable moving volumetric captures. Apple itself uses the technology for the spatial scenes feature of visionOS, and it's behind the massive upgrade in visual realism we saw from Personas in visionOS 26.\n\nRealityKit in visionOS 27 adds native support for rendering Gaussian splats, removing the requirement for developers to implement the code for this themselves.\n\n### Realtime Cloth Simulation\n\nRealityKit in visionOS 27 adds integrated real-time deformable-fabric simulation.\n\nDevelopers can create fabric that drapes under gravity, folds and wrinkles, collides with objects, responds to movement, and can be pulled or otherwise interacted with. The stiffness, friction, and material parameters can be set to change how the cloth behaves.\n\n### Immersive Audio Upgrades\n\nRealityKit on visionOS 27 adds a Reverb Mesh API, letting apps describe the geometry and acoustic properties of virtual environments.\n\nDevelopers assign materials to surfaces to specify how each affects sound, including frequency-dependent absorption, frequency-dependent scattering, and reflection and reverberation behavior.\n\nRealityKit then simulates how sound travels and reverberates through that modeled space, making audio correspond more believably to the VR environment you see.\n\nMany of these features were present in Meta's XR Audio SDK for Quest headsets, which the company put \"on freeze\" earlier this year. Valve's Steam Audio SDK also has many of these features, and Valve continues to update it.", "url": "https://wpnews.pro/news/visionos-27-is-a-much-bigger-deal-than-apple-s-keynote-suggested", "canonical_source": "https://www.uploadvr.com/visionos-27-announced-apple-vision-pro-wwdc-26/", "published_at": "2026-06-15 03:56:52+00:00", "updated_at": "2026-06-15 08:48:07.733869+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "computer-vision", "ai-agents"], "entities": ["Apple", "Apple Vision Pro", "Google", "Gemini", "Meta AI", "ChatGPT", "Claude", "Grok"], "alternates": {"html": "https://wpnews.pro/news/visionos-27-is-a-much-bigger-deal-than-apple-s-keynote-suggested", "markdown": "https://wpnews.pro/news/visionos-27-is-a-much-bigger-deal-than-apple-s-keynote-suggested.md", "text": "https://wpnews.pro/news/visionos-27-is-a-much-bigger-deal-than-apple-s-keynote-suggested.txt", "jsonld": "https://wpnews.pro/news/visionos-27-is-a-much-bigger-deal-than-apple-s-keynote-suggested.jsonld"}}