Generative AI for Mixed Reality and Productivity

Generative AI for Mixed Reality and Productivity

Published on October 1, 2024

Published on October 1, 2024

Exploring New Features in the Fluid App Beta

Our biggest release v0.22. Is now live & we’re so excited for you to try it out. In this post, I'll walk you through some of the most notable additions to the app.

Flow Typing

One of the most significant new features is what we're calling "flow typing." This is not a reinvention of speech-to-text technology, but rather a re imagining of how speech-to-text is activated within the app. Instead of having to click into a window and press a button on a keyboard, you can now simply pinch your fingers and say what you want to appear in the text box. When you release your pinch, the text is immediately inserted, and buttons appear allowing you to undo or continue.

For example, if I pinch and say "cool pictures of dogs," then hit enter, it will take me directly to a search results page for that query. This small user interface change significantly increases the speed at which you can navigate the web in VR.

Momentum Scrolling and Web Interaction

In addition to flow typing, we're also rethinking how users interact with the web in VR. One quality-of-life improvement that was recently added to Fluid is momentum scrolling. By pinching and pulling down, you can initiate a scroll that continues until you pinch again. This makes scrolling much easier than it was previously in the app.

Another new feature is the ability to use an assistant, which we're calling "V," to navigate the web. You can activate V by making a V-shape with your fingers. For instance, after making and V shape of my first two fingers, I can say "go to Spotify" or "go to Facebook," and it will open those websites in a new window. I can also use commands like "open a new window" to quickly create additional browsing spaces.

Tranquil Grandeur: A New Environment

Perhaps the most visually impressive addition to the Fluid app is a new environment called ‘Tranquil Grandeur’, also known as the Fluid Lodge. This hand-built space, created by our artist Camille, is not only beautiful but also quite large. It features several seating areas, each marked by a dot, that you can quickly navigate between by clicking on them.

What's particularly useful about this setup is that your windows will follow you as you move between seats. So, for example, if you have a window open at the desk area overlooking the stunning skybox, you can easily move to a spot in the living room to hang out with a friend, and your window will re-appear next to you in the same relative position.

Customizable Skyboxes with Generative AI

Speaking of skyboxes, another exciting feature is the ability to customise the world outside the windows of the Tranquil Grandeur environment (as well as the Cabin environment) using generative AI. By clicking the cloud-shaped button or using the "/sky" shortcut (just type ‘/sky’ in the browser tab), you can access the Fluid Skybox interface.

From here, you can describe the view you'd like to see, such as "the view from the top of a hill in a Northeastern Seaboard forest with a beautiful sunset," and hit enter. The app will then use Generative AI to create a unique skybox based on your prompt. You can even adjust the rotation of the skybox to view the generated world from different angles.

Customisable Paintings and Easter Eggs

In addition to the customisable skyboxes, we're also working on making the paintings on the wall customisable. Pretty soon, you'll be able to personalise these artworks to your liking, further enhancing the sense of ownership and immersion within the Tranquil Grandeur environment.

We're really proud of the art in this space, and we've put a lot of effort into ensuring that the textures are high-quality and the environment feels hand-crafted. In fact, everything you see in here was hand-built by our talented team.

As you explore the Tranquil Grandeur environment, keep an eye out for hidden Easter eggs that pay homage to various members of our community. We won't spoil the surprise by revealing where they are, but we encourage you to take your time and see if you can spot them all.

Conclusion

The new features we've added to the Fluid app, such as flow typing, momentum scrolling, the "V" assistant, and the Tranquil Grandeur environment, are just the beginning of what we have in store. We're constantly working to improve the user experience and push the boundaries of what's possible in VR.

We're incredibly excited for you to try out these new additions to the Fluid app and to hear your feedback. Your input is invaluable as we continue to refine and expand the capabilities of our platform.

Stay tuned for more updates, and as always, thanks for being a part of the Fluid community.

Download the Fluid App today for free.


Instead, we’ve come up with our own approach to mixing the real and virtual world that we think is both simpler and better tuned for our use case. We’re calling it Reality Blending. You can watch the video at the end of this blog post to see what it looks like.

To activate reality blending from full immersion, just click the passthrough button on the dock. Your desk surface and the area around you will become visible and seamlessly blend into the virtual world around you. We have also added what we’re calling Reality Blending Modes, which are custom shapes for the passthrough area. Today we’re shipping two Reality Blending Modes, Front and Surround.

In the Front Reality Blending Mode, the area in front of you is visible but your peripheral vision sees virtual content, which enhances focus and immersion without limiting visibility of your desk or your lap.

In the Surround mode, you are fully surrounded by an area of passthrough, which stays beneath you so you can see your desk, seat, and any obstacles or objects around your feet and waist. You can choose which of the two blending modes you want to set as the default in the Fluid Settings App.

You can always switch to full passthrough by clicking the passthrough button, or back to immersive mode if you don’t want to see any of the real world (which also comes with a boost in CPU performance.)

We’ll be introducing more Reality Blending Modes over time based on user feedback, so please give them a try and let us know what you think in our Discord. You can try Reality Blending today in the beta version of Fluid. You can find instructions on how to get the beta version of the app on our Discord.