Designing for Immersive Art Experiences
VR · Multimodal Interaction · Sound Design
InMuse — A VR museum with an AI guide
Step inside a painting, with multimodal interaction and sound design built around when the guide speaks.
Role
Sound Design · System Architecture · Multimodal Interaction
Duration
Mar – Jun 2025
Scope
5 modality channels · 8 interface sounds · 3 background tracks
Team
Conversation Designer · Visual Designer · UX Researcher
The prototype walkthrough, at 1.5× speed and without sound: the lobby, the radial filter, an artwork’s detail, and Begin.
Three scenes, one continuous conversation.
The concept film follows a single visit from arrival to reflection. It is the only honest way to judge the thing we actually designed, which is timing — when the assistant speaks, how long it waits before it does, and what the interface sounds like when nothing is being said. Worth playing with sound on.
Four features carry the concept. The environment is built from the artwork itself, with 360° visuals, sound and gesture. An AI companion gives context about the artist, the period and the technique when you ask for it. Visitors leave spoken or written notes for each other. And the whole visit works for someone who cannot travel to a museum.
Arrival. The assistant greets a returning visitor by name, asks what they came for and whether they want a guided or a reflective visit, then suggests artworks to match. The intention is set in conversation, not in a menu.
Inside The Starry Night. The visitor moves by voice and gaze, and the assistant offers context only where attention has already settled.
Reflection. When the visitor pauses, the assistant offers to take a note, can add context such as something Van Gogh said about the painting, and brings up notes that earlier visitors left in the same place.
The Problem
A museum asks you to come to it — at its hours, at its pace, in its crowd.
Standing in front of a painting is meant to be an intimate act, and a gallery is one of the least intimate places to do it: a crowd, a rope, and wall text written for everyone. There is no way to ask the obvious question, and no way to tell anyone what you noticed. We designed for two groups. Art enthusiasts and students want time alone with a work, context they can ask for, and other people to talk to about it. Visitors who cannot easily get to a museum, because of mobility, health, distance or cost, need the visit to come to them.
The visitor we designed for first rather than last. Remove travel, stairs, crowds, standing and small type in low light, and what is left is the part that was supposed to matter.
In VR, every piece of interface you add is something standing between the visitor and the art.
On a screen, UI is how you reach the content. Inside a painting, UI is the thing blocking your view of it — a menu bar floating over The Starry Night does not help you see it, it reminds you that you are wearing a headset. So the brief inverted. Instead of asking what the interface should show, we asked how little it could show and still be usable. Voice where a menu would go, radial wheels for filtering and settings, sound instead of labels, and an assistant that stays quiet by default.
The Lobby
Before you are inside anything, you have to choose — and that is the only place a panel is allowed.
The lobby is the one part of InMuse that behaves like software, because browsing a collection is a software problem. You can browse by voice, by gesture or with the panels. The headset handles the input, and the cloud holds the collection and filters it by theme, time and era, artist or popularity; the panels, the interface sounds and the ambient bed are stored on the headset. Filtering happens on a radial wheel rather than a dropdown, because a wheel has no scroll position to lose and can be turned without looking straight at it. The lobby is also where you search, reopen past sessions, and keep artworks in My Collection, albums you put together the way you would a playlist. Once you press Begin, these surfaces go away and one small menu is left: turn the assistant or the background sound off, and view or leave notes. When you leave a note, the assistant can organise or summarise it for you.
Left to right: the collection, the radial filter open over a filtered set, and the last screen before immersion. Everything here is translucent on purpose — the lobby is a room you pass through, not one you furnish.
Five channels to carry meaning, so that no single one has to carry all of it.
Rather than designing screens, I mapped every task across the channels a headset can actually use — visual, audio, gesture, haptic and olfactory — and decided per task which channel leads and which stays silent. The map became the spec. It is the thing that stopped us defaulting to a floating panel every time something needed to be said:
01
Visual carries state, never instruction — a highlight, not a sentence.
02
Audio carries confirmation, so the eyes never have to leave the artwork to check that something worked.
03
Gesture and voice carry intent, which keeps both hands and the whole field of view free.
The modality map for one of the three phases. Rows are tasks, columns are the five channels a headset can use; green is voice input, grey is gesture, outlined is an output cue. Three of these exist — Into the Art, Inside the Art, Community + Closing — and together they are the spec that stopped us reaching for a floating panel every time something needed saying.
When you cannot use a label, the sound has to say what the thing is.
Eight interface sounds and three background tracks carry the parts of the interface that have no visible surface: navigation, filtering, selection, entering and leaving an artwork, and the failure state for a gesture that did not land. Each one had to be identifiable without being looked at, short enough not to collide with the next action, and quiet enough to sit underneath the background track rather than fight it.
The selection wheel is the clearest case. It could have been a click. Instead it is the sound of a brush being lifted from a palette, because the visitor is standing inside a painting and a click would have put them back inside an operating system. That sound is doing the work a label would do on a screen — without taking a single pixel away from the artwork.
A single soft swell. It marks the boundary between the room you are in and the one you are entering.
Background sound
The bed under every menu move. Low and continuous, so moving through the interface never sounds like a decision.
A light tick per filter. Confirms the hit at the moment of contact, which is what a hover state would do on a screen.
A brush lifted from a palette. The signature sound of the project — it says “you are inside a painting” where a click would have said “you are inside an operating system”.
The generic confirm, kept deliberately plain so the two brush sounds stay special.
Two themes, dreamy and meditative. Both, and the background sound, can be turned off from the in-artwork menu.
Where a row has a button, it plays the file from the build. The two long beds are 22-second excerpts; the interface sounds are whole.
The Look
Dreamy. Serene. Future. Soft. Fluid.
Five words settled the visual direction before a single screen was drawn, and they had a job: a virtual gallery has to be neutral enough that the art is the loudest thing in it. So the panels are transparent, the controls are minimal, and the type scale is large enough to read in a headset while the secondary text stays deliberately quiet. The ambient scores were chosen against the same five words, which is why the sound and the look agree without either one having to reference the other.
The team’s moodboard, the source of the five words above. The paint in the middle is the one thing the interface is never allowed to compete with.
The design system from the final deck. Type is set in Inter, from 33 px for primary text down to 12 px for body and button text, on translucent panels that let the room show through. Two radial wheels do the work a menu would: Filter by in the lobby (theme, time and era, popularity, artist), and Settings inside an artwork (turn off the AI agent or the sound, leave or view notes). The serif labels name the sound each control plays, the same events as the sound palette above.
Why split it at all?
Because a voice that lags is a voice you stop trusting.
Speech recognition, gesture recognition and text-to-speech all run on the headset, because a pause between speaking and being heard reads as the system ignoring you. Two consequences fell out of that split and both turned into design decisions. The first is a privacy one: your voice never leaves the device — only a transcript of it does — which is a sentence worth being able to say out loud to a user. The second is bandwidth: the 360° visuals stream from the cloud and are enhanced on the headset for smooth motion. The soundscape works the same way, assembled on the headset from stored components against instructions the cloud writes for the chosen artwork.
The system architecture from the final deck: what runs on the headset and what runs in the cloud, coloured by subsystem, with what passes between them written on the arrows. The local column is everything that would be unforgivable with a delay.
Shown as a working concept, and picked apart in exactly the right place.
The demo was shown and critiqued rather than user-tested, so there are no numbers on this page. Two things came back consistently. The immersion and the sound design landed. The spoken narration did not — more than one person said the commentary would be better delivered some other way. That is the most useful sentence anyone gave us, and it is why the last section exists.
Limits
The honest problem with this project is that VR is the barrier it claims to remove.
The case for InMuse rests on access — a museum for people who cannot easily get to one. But a headset costs money, needs a certain amount of technical confidence, is uncomfortable for some people to wear, and is its own accessibility problem. That is an immersion-versus-access tradeoff sitting at the centre of the concept, not at the edge of it, and the honest version of the next step is a phone build that gives up most of the immersion to reach far more people.
Two more we could not design our way out of inside the sprint. Adapting an artwork into a walkable environment is an interpretation, and it needs the consent of the artist, the estate or the museum — we used The Starry Night precisely because it has already been adapted into an immersive exhibit, which sidesteps the question rather than answering it. And a museum may reasonably read this as competition rather than a companion; that is a partnership problem, and no interface decision fixes it.
The brief listed two further risks. Community notes would need moderation, which was out of scope for the sprint. Artists may also want to make work for this medium instead of having existing work translated into it, which would need tools for artists that we did not design. Under all of these sits the first rule we set ourselves: a multisensory version has to respect the original work and what the artist meant by it.
The critique pointed at an assumption none of us had noticed we were making: that meaning has to arrive as a sentence.
01
Put the commentary where the visitor is looking, not in their ear. Spatial audio anchored to the part of the canvas it describes turns narration into a property of the room. You decline it by turning your head, which is a far gentler no than reaching for a mute button.
02
Let the artwork do the pointing. A half-stop of light on the region the assistant would have described, or a brushstroke that animates for a second, says “look here” without a sentence and without a voice. It also gives deaf and hard-of-hearing visitors a cue that voice never gave them.
03
Score the commentary instead of speaking it. The sound palette already proves meaning can travel as timbre — the brush lift says “you are inside a painting” faster than any label could. The same logic can carry context: shift the instrumentation and density of the score rather than narrating what the visitor ought to feel.
© 2026 Sam Wang