Spelunking WebXR

My work this week began with a deep-dive on WebXR. WebXR is super cool, but the open source scene around it is struggling, in a cyclical pattern with device and browser makers of disuse.

• Mozilla made massive early investments, but retreated, leaving behind an abandoned WebXR Viewer for iOS app and open source support, like a Unity exporter that doesn't work anymore.
• As long as Apple keeps WebXR support out of iOS, experiences can't rely on it, but VR headsets weren't worth buying until a year or two ago, and they're still not for many consumers. This feeds the cycle of there not being content produced, developer demand to improve the platform, or platform improvements.
• Apple has shipped a beta of WebXR in Safari on visionOS, but there are clear bugs and compatibility issues between "natural input" with eyes/hands vs the controller system on other headsets, and the development tooling is next to nonexistent. Apple's WebKit team put out a post showing off natural input with WebXR, but failed to open source their example project, so there's no clear path for developing with it.
• Privacy restrictions on headsets, especially Apple Vision Pro, severely restrict what can be built, especially with WebXR over the native tooling. There's no camera input for placing objects automatically, or support for passthrough video manipulation with shaders etc, which I was hoping for adding some subtle lighting effects with. This makes sense as opening that door will create dystopias we can barely imagine, but is creatively limiting.
• The stock UI kit of visionOS, while simple to work with in SwiftUI, looks gorgeous. The WebXR platform begins with primitives of 3D shapes, and doesn't interplay with DOM, so building 2D UI controls is challenging, and making the accessible is even harder. The closest comparison in WebXR is pmndrs/uikit, which looks far cheaper.
• For privacy reasons, WebXR UI elements have no access to gaze tracking hover effects on visionOS either, so dense interactive UIs aren't feasible. Hands are sometimes not cut out to be shown while in web-based VR environments, which is super disorienting when eye tracking controls don't work either.
• I couldn't find examples of AR instead of VR WebXR. Not having the passthrough option, and needing to define the look of the entire virtual space, raises the pressure on developers and makes for worse experiences in most cases. The number of dialogs to accept (enter VR from the site, system dialog for hand tracking, system dialog for motion tracking, system informational dialog for an immersive experience) make entering tedious.

My project this week collapsed under its own weight. Originally conceived to combine WebRTC live video in React with a 3D space in WebXR, then embeddings on live data in 3D space, combining several technologies I've never used, I whittled it down until getting stuck on technical details that didn't work out in my timeframe.

I reworked my backend from last week to map my coursework into three dimensions instead of two and color-code the cards by course, which was successful. However, when it came to displaying the cards, using the rickety UI of uikit I struggled to place in 3D space.

WebXR is simultaneously far more developed than I knew, especially in frameworks like aframe, and more limited than I was hoping, offering a fraction of the quality, accessibility, development ease, onboarding, and performance of native app immersive experiences on visionOS.