Hand Tracking Innovation Colocates Vision Pro & Meta Quest Mixed Reality

EXCLUSIVE: I tried a groundbreaking new idea for hand tracking that successfully syncs the experience of people in headsets from Apple and Meta.

A first-of-its-kind app available today on Apple Vision Pro offers a groundbreaking new idea for colocating people and their experiences in VR.

From today puzzlers can piece together in the same shared reality on Apple Vision Pro and Quest with a new trick developed to simplify colocation. Deployed by Steve Lukas in his app Jigsaw Night, the method sees a person wearing a headset put their hand up in front of another’s headset to create a bridge between their devices. If you’ve ever worn a headset and accidentally seen it track someone else’s hand instead of your own, Lukas’ feature uses that quirk to share the relative position and rotation of a virtual object to effectively sync the realities of both headsets.

“Part of the process of porting Jigsaw Night to Apple Vision Pro was to port the Meta Interaction SDK, which has a consistent method of tracking hand joint poses. So we already had a system that essentially read hands in the same way even though the underlying architectures may be different,” Lukas wrote. “When puzzling together with friends and we got too close together, we noticed that our hands would sometimes get ‘grabbed’ by the wrong devices under a specific set of circumstances. So…if we purposely reconstruct those circumstances, can we have a situation where both headsets are tracking the same hand, each thinking ‘that’s my hand’ and being able to use that as a solid ‘shared anchor’ in 3d space? The answer was yes.”

The app features a circular table that functions as a control center to change options and the gesture syncs its relative position and rotation between the two virtual realities. By repeating the process to other headsets, several players piecing together a jigsaw puzzle together in one physical location can create one single shared space with remote players.

“Really it could be done with any headset,” Lukas texted, teasing that it should be possible to “auto-sync with hands between Quest, AVP, and other immersive platforms and view it all through a phone or iPad.”

Most multiplayer VR games don’t allow players to hand digital object to one another remotely and the number of apps that enable the same thing with someone standing in the same physical room is vanishingly small. The app Figmin XR, for example, allows players to sync their realities but you have to carefully put a shared anchor in the same spot on the floor on each headset and ensure it’s at the same height with the same rotation.

In April, I outlined colocation as the biggest problem to solve in software for mass market VR after I watched Quest headsets struggling to consistently create this experience with shared spatial anchors at an event in Atlanta. Since then, Lukas has attacked the problem in every possible direction digging into Unity’s ARFoundation, Google’s ARCore, Apple’s ARKit, Meta’s Shared Spatial Anchors and even developing his own Local Area Network techniques to more robustly allow devices to share reality. Early this week I was the first to test the feature ahead of the launch of the Jigsaw Night companion app for Vision Pro and iOS.

You can find Jigsaw Night on the Meta Quest Store and Apple App Store and find guides to the companion apps in headsets and phones via the game’s website.

This publication is a 100 percent independent, community-supported journalism effort made exclusively by people for people. Please consider a donation to support our reporting or become a paid subscriber.