There’s one thing all mixed reality headsets (and some VR headsets) have in common. But this technique has yet to really perfected. This is the “passthrough”. A concept made possible by a meticulous arrangement of cameras and lenses. It’s thanks to this concept that headsets like the Meta Quest Pro, Apple Vision Pro and even the Quest 2 can simulate your physical surroundings – to varying degrees – while covering your real view.
The “passthrough” allows the operation of augmented reality applications for interior design and interactive learning. Use cases that involve some degree of spatial awareness in the real world, and significant computer processing for image simulation.
Naturally, it is not easy to get a visual experience similar to what one has in real life. Especially if we take into account the distance between the location of the cameras of the helmets and that of our own “cameras”, ie our eyes. But Meta has a secret weapon hidden within the Display Systems Research (DSR) team at her Reality Labs which she says has managed to make a big breakthrough in the field. of the pass-through.
Light Field Passthrough, inspired by insects
Here is Flamera, a mixed reality headset still in pre-production that uses “Light Field Passthrough” to accurately reproduce your environment. How ? By adding a grid between lens arrays and camera sensors. Enough to obtain optimal light capture, regardless of eye movement, and a natural recording of the distance and movement of objects around you.
Flamera pre-production unit. Meta
This aperture module gives what can best be described as the appearance of a bug’s eye, with holes pointing in all directions and from all angles. The headset takes the raw sensor data captured by all those holes and reconstructs it through depth-based reconstruction, according to Meta. Below you can see a comparison between the “passthrough” mode and the real thing.
“Flamera’s optical design works best when the headset is thin, allowing us to place the passthrough cameras as close to the user’s eyes as possible,” says Grace Kuo, a researcher with Meta’s DSR team.
Last month, the Flamera helmet was demonstrated at Siggraph 2023, an annual conference on interactive techniques. “This was hands down one of the highest quality augmented reality passthroughs I’ve ever seen,” said Anshel Sag, principal analyst at Moors Insights & Strategy.
Sag also pointed to the beta headset’s reduced field of view, which can be a problem for VR/AR applications that require side-to-side looking.
Still, Flamera could be the start of something big. And the list of new mixed reality headsets to be released in the coming months, culminating in the launch of Apple’s Vision Pro early next year, makes technology developed by the DSR team of Meta an item to watch.
Source: “ZDNet.com”