XR Experiments

2022   ·   The New York Times   ·   AR   VR   Mixed-Reality   Unity   WebXR   Web

Through 2022, our R&D team ran a series of experiments probing how The New York Times might tell stories in mixed reality, first natively in Unity on the Meta Quest, then in WebXR.

👤 Role: Software Engineer, Ideation, UX Design, Prototyping

Spatializing the Space Section

Presenting The Times in mixed reality calls for a new paradigm for discovering and exploring stories. What is the immersive version of a section landing page? For this experiment I built a spatialized version of the Space & Astronomy section in Unity for the Meta Quest headset, where readers navigate its stories anchored in a 3D solar system rendered in passthrough.

The idea started on paper. A reader taps a 3D “T” to wake the app into an iPad sized feed pane, swipes from a Top Stories menu into a section, and is met with a 3D representation of that topic, a globe for World, the solar system for Space. Stories are pinned to their location in space. Selecting one transitions from the windowed view into a full dome, matching the level of immersion to the reader’s deepening engagement. The prototype included content from previously published articles Exploring the Solar System, 100 Images From Cassini’s Mission to Saturn, and Mapping Saturn’s Moons

Storyboarding the flow: desk → section → 3D scene
Exploring the solar system, story by story
Modeling the 3D “T” in Gravity Sketch
The finished 3D “T” app icon

The prototype ran in Unity on the Meta Quest using passthrough, used the Oculus Interaction SDK for hand tracking (grab, drag, pinch, and poke), and DOTween for animation. The scene was built from NASA 3D assets, SPICE mission data, planet textures, background stars, and probe models. Article text, scene state, and links were all driven by JSON so content could be wired and tuned without rebuilding. The 3D “T” icon was modeled in Gravity Sketch.

Captures

Tapping the icon reveals the splash and menu; swiping from Top Stories into a section reveals its 3D scene.
Swiping into Space, the sky fills with stars; the background fades with speed to avoid motion sickness.
Stories are selected by pointing at a pinned marker and pinching, or from the menu.
Selecting a story fades the windowed view into a fully immersive dome.
Left/right arrows move between articles; the up button returns to the section menu.
Links inlined in the body text can be tapped to jump to a related story.
Transitioning between the framed “window” view and full immersion.
The Earth asset has independent day/night and cloud layers.
Tapping the X closes the story and section and returns to icon mode.
Cassini’s flight path, converted from NASA SPICE data into a path animation in Unity.

The design patterns here could extend to other sections of The Times, world politics via a globe, local news via a map, health via a body, sports via a stadium, or books via a library.

Formula 1 in Unity

A follow-on experiment bringing the story of the Formula 1 race car redesign into mixed reality in Unity.

Experiments on the Immersive Web

A second wave moved from native headset apps to WebXR, prototyping spatial scenes and hand interactions that run in the browser on a headset with no install required.

Empire State Building

An MR reimagining of Why the Empire State Building, and New York, May Never Be the Same in which you can “unzip” the iconic building to see inside.

👥 Credit: Ambient city sounds by knufds, Ambient office sounds by pixabay, other Sound effects by kenney

Yellowstone

An MR reimagining of Your Children’s Yellowstone Will Be Radically Different where dragging the terrain advances the story and the seasons. Assets prepared in Q-Gis. Tree’s algorithmically placed based on USDA Forest Service data.

👥 Credit: Terrain color: Sentinel-2 WMS, cloudless layer for 2020 by EOX - 3857, Elevation: 3DEP WCS, 3DEPElevation, Tree density data: MRLC WCS, NLCD 2016 Tree Canopy L48, Tree model by kenney

Grab gesture and terrain clipping test

World Cup

An MR reimagining of Richarlison, Messi and Pulisic: Three Stunning Goals Frozen in Time.

Poke interaction test