Apple has received or applied for patents to enhance the Vision Pro by turning flat surfaces, such as desks and interactive touch-sensitive displays, into touchscreens. This technology solves ergonomic issues of in-air typing by using thermal touch or computer vision to detect real objects. Touches are translated into virtual commands.
Key Aspects of the Technology
- Surface mapping: Vision Pro recognizes surfaces and places apps or controls, such as a keyboard, directly on a real desk.
- Thermal touch technology, from Metaio (acquired by Apple in 2015), uses infrared sensors and thermal cameras to detect heat from a user’s finger when it touches a surface, turning that touch into a command.
- Virtual trackpad/input column: The patents describe enabling any flat surface to function as a virtual “magic trackpad,” enabling gesture input without an actual device.
- Developer applications: this foundation enables apps such as note-taking tools like Touch Desk to run in the background and let Users jot notes on their desks. It also boosts productivity and helps the system recognize when a hand covers a virtual object.
- Alternative to the Vision Pros in-air virtual keyboard: This technology offers a practical alternative that addresses the lack of haptic feedback during long typing sessions.
This technology is part of a broader spatial computing ecosystem intended to seamlessly integrate physical environments with the digital world. Interaction models.
An Apple patent granted last week described a wide range of potential Vision Pro accessories. Notably, it details a hardware device that turns your desk dash or any flat surface dash into a virtual magic trackpad with full gesture support, enabling more versatile and immersive interaction with the headset.
At first glance, the patent is somewhat odd: it uses one piece of physical hardware to emulate a virtual view of another piece of physical hardware. However, despite the initial strangeness, there are some potential benefits to Apple’s approach.
Vision Pro Accessories
Building on this, the more general patent describes a modular approach to adding hardware capabilities to a headset like Vision Pro.
These include additional cameras for an even wider field of view and a range of sensors to enhance the headset’s capabilities.
Given Apple’s strong health focus, it’s not surprising that some of the proposed accessories are health sensors of various kinds. Apple further describes fashion accessories.
Virtual Trackpad
With this in mind, the most exciting possibility for this technology to me is replacing a Mac and an external monitor with a headset, whether for travel or permanent use.
To try out similar solutions, I’ve been experimenting with a Meta Quest app that lets you run multiple virtual Mac monitors. I’ll write more about it in a separate piece soon. While controllers and hand gestures work, they are no substitute for a magic keyboard, which is why I’ve been using the headset with a physical keyboard and trackpad.
Apple’s proposed approach addresses this by potentially turning any flat surface, from a desk to an airline tray table, into a virtual trackpad with full gesture support.
While this concept may be possible using vision‑pro cameras to detect hand gestures, the patent notes that this method may not always be reliable; as an alternative, it suggests that cameras in external devices placed on the surface could perform better.
Notably, a device may be better able to detect surface taps because it is also located on the surface, and therefore, sensors may have a clear line of sight to the tap location. In contrast, [another] device may resort to depth analysis to determine whether the object has moved along the z axis sufficiently to qualify as a tap on the surface in some embodiments. The set of one or more criteria includes a requirement that the object be valid. For example, an object is valid when it is a digit of a hand. In some embodiments, an object is a valid object when the object is a writing instrument. (e.g., pen, pencil) or a stylus.
But the gist appears to be that a camera on a flat surface will be better at detecting a gesture, like a trackpad tap, than a camera mounted on the head.
The patent illustrations show a small box on the table that detects other Magic Trackpad–like gestures, such as rotating a photo with the thumb.
What’s the Benefit Over a Physical Trackpad
If the hardware only emulates a trackpad, why not use a regular one?
The patent doesn’t address this directly, but the illustration suggests the device may be smaller than a typical trackpad. Additionally, since it tracks both a writing instrument and a hand, the accessory might offer greater flexibility than a standard trackpad, possibly allowing users to interact in more ways or adapt to different input needs.
Will We See Vision Pro Accessories at Launch?
With significant time before launch, Apple still has room to introduce new Vision Pro accessories or hint at future models, potentially shaping the user experience in innovative ways and keeping anticipation high for the upcoming release.
Source: Vision Pro accessory could turn any flat surface into a virtual trackpad, with gesture support
