Journey Into Self
3D ANIMATION | MOTION CAPTURE | LIVE DANCE PERFORMANCE
TECHNOLOGY: UE5 / Oculus / Stable Diffusion / Blender / Houdini / Figma
TEAM MEMBER: Shimin Gu /
ROLE: Interaction Designer / Software Developer
EXHIBITION: Shanghai International Film Festival XR 2024 / Galerie Met Berlin Exhibition 2023 / :iidrr Gallery NYC Exhibition 2023
“Journey Into Self” is an immersive VR experience based on theories from cognitive science and personal inspiration. It deconstructs the abstract self into a four-layer spherical virtual space resembling a spiral gallery.
In each layer, it gradually constructs shapes representing time, space, emotions, cognition, and beliefs within the inner world. Viewers, adopting a first-person perspective, accompany protagonist Anne as she tours the gallery, jumping to different time points in her life and interacting with Anne at various ages to help her rediscover and understand herself.
While experiencing Anne's life, the piece also offers viewers a fresh domain and perspective to contemplate the self, inviting them to use the structure of this virtual space as a reference to create narratives about themselves.
Design Overview
One of the challenges we faced was that most props in the experience are interactive and respond to users' actions, leading to different environments and visual effects. As a result, animations often cannot be pre-animated but must be controlled by code. This requires extensive consideration and testing to ensure smooth performance.
Card Selection
NPC Interaction
Onboarding: The very first scene users will be landed. A set of cards introducing Ann’s background story and explaining Emotional Spheres’ connection with people’s stories and experiences.
Time: The installation serves as a time selector. Users can interact with the rings to select different timestamps or periods from December 2019 to January 2023.
Space: The space floor has seven huge floating spheres with beautiful visuals. The story and scene inside each portal are based on user input.
Backstage: The Backstage level is a place for emotional reflection. The installation on the ceiling takes over the entire surface and reacts to the sound.
Theater: This floor has a small pond, with a rock in the middle. Users will be led to draw on the water surface and navigate to the center. The stone will disappear after interaction which is the end of the experience.
Interactions
Time Selection
Dynamic Objects and Lights Follow
Enter Emotional Balls
Interactions
Drawing on Water
The most challenging part for me was simulating drawing on water. While there are multiple ways to achieve this, making the strokes look beautiful and naturally dissolve into the water so that users feel they are drawing on a water surface, rather than in the air, is particularly difficult. Additionally, other animations needed for storytelling depend on the drawing process, and these effects can be resource-intensive.
After extensive exploration, the final product features a small pond made of several layers. The first layer is a plane with simulated water materials, while the second layer handles water simulation using Unreal Engine’s water plugin to simulate forces on the water. The physical water material detects user input vectors through line tracing. The strokes are created using a Niagara ribbon particle system, with changes in color, size, shape, and opacity over time to achieve the dissolving effect.
explorations
In this project, we experimented with various new technologies to achieve our goals. We used Stable Diffusion to generate visuals for the cards, Elevenlabs to train voices for NPCs and voiceovers, Move.ai and Livelink for motion capture, and the ChatGPT API to generate real-time AI responses. While not all of these technologies made it into the final product, they were valuable and enjoyable explorations.
The project is still ongoing, we wish to turn it from a PC VR to a mobile VR experience, change all the controller inputs to natural gesture inputs, and introduce a XR part into it, so stay tuned to see how it evolves in the future.