Reality Blending: An elegant way to mix passthrough and virtual content

Reality Blending: An elegant way to mix passthrough and virtual content

Published on October 1, 2024

Published on October 1, 2024

At Fluid our mission is to make the web insanely great with spatial computing and we’re constantly experimenting and exploring to find better ways make the internet come alive in the space around us.

Mixed reality headsets like the Meta Quest 3 and the Apple vision pro use passthrough cameras to create the incredible illusion of seeing through your headset so you can see both virtual content and the real world at the same time. This is an awesome evolution for virtual reality because it lets you to enhance your reality with virtual displays, decorations, ai pets, and even other people who are joining you from far away.

Mixed reality apps fall on a spectrum between complete immersion and complete reality.

The challenging thing about developing mixed reality experiences like Fluid lays in how you blend the virtual and real world together to create a seamless experience. You can give users the ability to work in full passthrough, full immersion, or a mixture.

Fluid is the first standalone mixed reality app for people who want to primarily use a keyboard and mouse to control standalone spatial computing headsets, so we really care about giving users who mostly want to be immersed in their work a great partial passthrough experience that still lets them see the keyboard, mouse, and coffee cup on their desk.

Up until now we used a rectangular window with a couple of control buttons that let you hide it or unlock it so you could move it around in space. Other productivity apps even adapted this design because it gave users a lot of flexibility to change their workspace up in the moment. However, we were never quite happy with this solution and we’ve been searching for something better.

Apple vision pro shipped with a neat “digital crown” that lets you dial the amount of passthrough surface in your peripheral vision. We thought it was really well done. It feels smooth and premium, but at the same time it doesn’t quite fit the unique needs of people using their headsets as spatial computers because if you want to have your desktop or lap visible, you need to dial back the immersion a lot more than we would like.