「Empire at Sea」の技術革新――物語主導のミクストリアリティ気候寓話

The Voices of VR Podcast. Technical Achievement Award at Vanessa Mercer. So Empire at Sea was probably one of the most impressive technological, awe-inspiring experiences that I've seen in a long, long time. So just to set the scene, you walk in, there's this big table that's like 8 foot by 14 foot or so, and there's 10 Apple Vision Pros, and it's got these sort of octagon LEDs, so that when you put on the Apple Vision Pro, you end up seeing what is projected onto it, the 4D Gaussian splats on top of like a Gaussian splat capture of the locations.

And so also the LEDs are sort of like as the characters are moving around, they're kind of lighting up. And so it gives this kind of like Star Wars holographic effect. And the fact is they're kind of blending together those, just this amazing kind of layering. So that is a whole visual spectacle within its own right. But the audio is also really amazing because they have speakers in the room, but they're also using the Apple Vision Pro and they're kind of augmenting different aspects of blending the audio. So you're hearing a big cinematic orchestral mix in the background, but also hearing dialogue. And the dialogue changes based upon where you're moving around. And so in the different parts of this experience, based upon where you're at, you can basically hear different aspects of the story as they're unfolding. There's one part when there's like more of a multi-threaded piece where there's like four or five different rooms with different actors that are all talking about a similar theme. And so this is kind of a speculative features piece. They're on this oil rig, and there's a scientist here. And they basically have these visitors that are coming. And are they a threat or are they going to be welcoming in and they have other ways that's going to benefit from having this diversity of people coming in this kind of immigration context and this kind of speculative future?

And so it's also a bit of a parable of thinking about climate futures in terms of climate change and what does it take for us to be in relationship to each other as we move forward. And it's such a really powerful story, the way that it has this dramatic moment at the end in these speeches that they have to make this decision as to which way they're going to go. And so it just ends up being a really amazing and evocative piece, a really impactful story. It was my favorite piece that I saw this year at Venice Immersive just because of the technical achievement, but also just really resonated with the story that they're telling. Again, this is a story that was being supported by a GOG, and so I think there's a lot of these climate change and the impact of using XR to be able to tell these impactful stories. And so in terms of the elemental balance, it's Earth-Water. So the Earth element being so much of it in this kind of mixed reality, and there's so much around the physical installation of this, the table and just being in this space with other people. You do have a bit of a shared experience, but also very driven by the story and the narrative. So the water element, it's this multi-threaded and story-driven piece that, yeah, is actually using potentially the Intel Capture Studio.

Peter could neither confirm nor deny, but basically the time and place when this was shot, there was basically no other real option for how this could be shot. So back in episode 1790, I talked to Sarah Vick of the Intel Capture Studio back in 2019. And this is an unpublished interview that I just published as a part of this larger series. You can go back and kind of listen to what was happening back in 2019. So it kind of fits in the time and place when this kind of large-scale volumetric capture was available. Interestingly enough, I think the Intel Capture Studio shut down in like 2020. It lasted for a few years. It didn't really last through the pandemic. But now that the 4D Gaussian splats, I think it's worth noting that the top three pieces all used 4DV.ai of this volumetric capture of Gaussian splats. And so even footage that may have potentially been captured with Intel Capture Studio, having multiple people being captured at the same time, that was something very unique that doesn't really necessarily exist today. but I expect to make a comeback based upon the experience that I'm seeing here because there's going to be a need to be able to do large-scale volumetric captures

of lots of different people interacting with each other and to try to composite the timing and all the different nuances of that is just too difficult to do when you're in these small capture spaces doing one at a time. And so a piece like this really demands to have lots of people being captured at the same time. And so 4D Gaussian Splash have kind of innovated so many different aspects of both the quality level of volumetric capture, but also the compression to be able to have it run locally on Apple Vision Pro and 5s to be able to actually render it locally without having to do the additional latency that would happen if you had to do it rendered in the cloud. So being able to do that all locally is incredible, and I expect to see way more innovations moving forward with this kind of 4D Gaussian splats. And it's worth noting, again, the top three experiences at Venice Immersive were all using this kind of 4D Gaussian splat technique. So expect to see a lot more volumetric capture here in the future, especially in this high-end ability to be able to capture amazing acting performances with the emotional expression.

So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Peter happened on Friday, September 4th, 2026 at Venice Immersive in Venice, Italy. So with that, let's go ahead and dive right in. My name is Peter Flaherty. I'm a director and a creator in the XR space. I'm particularly focused on interactive and participatory narratives, and I'm interested in making dramatically and emotionally connected stories. Maybe you could give a bit more context as to your background and your journey into the space. Yeah, I started out making projections for immersive and interactive theater performances about 25 years ago, And I worked in the U.S. and Europe with groups like the Builders Association, Complicite, Basil Twist, the Puppeteer. And we did really inventive work with video on stage, sometimes as a virtual character, sometimes as the primary narrative driver. People often described the work as it looked like a film exploded onto a stage.

So we weren't really making plays per se, but we sort of took over theatrical spaces and activated them, in my opinion, with the kind of imagery and storytelling tools that this generation is accustomed to. So film and television language and figuring out how to connect that with live performance. So you had the sense of presence and liveness, but you also got sort of the vocabulary of the movies in a sense. great maybe you could give a bit more context for empire at sea what's the story that you wanted to tell about this piece so empire at sea has been in gestation for about eight years so it's an interesting project in that way we talked about my last project the dial my previous project at sundance in 2019 it was a tabletop drama for three simultaneous participants using iphones at the time But I was really interested in figuring out how to scale it.

And that project used projection mapping as one of the storytelling devices and an interactive light sculpture. And for Empire at Sea, I was sort of thinking, okay, will the technology advance to a place where I could use real-time photorealistic characters that were completely responsive so that I could put people in an open world? because it's my belief that characters that are on rails in an open world don't really function at full scale. I don't think it's a satisfying experience because I don't feel embodied alongside the characters if they're unaware of me in a space and pass through me. So I was thinking, okay, so do I find a technology that's able to do that? And there were a few that were emergent, game characters effectively and the sort of start of metahumans. but I still had this interest in using shifting scales, so a tabletop piece that then might become life-sized and back and forth, just as a methodology for storytelling.

So how do we frame this new medium? How do we create some storytelling guidelines or rules? And that's always helpful when you bump up against these rules because they create logic for your audience. so I was driving one day down the PCH in California I had been visiting a friend of mine up in Santa Barbara I live in Los Angeles and there are these decommissioned oil rigs off the coast and at sunset they're just really gorgeous because they're backlit by the sunset on the Pacific Ocean and I started looking at them and I was always curious about what happens with them after they're decommissioned So I had read that you could actually go out and stay on some of them. And so I was getting curious about them just on this drive. And I just thought, wow, that kind of looks like a table. Wouldn't it be interesting to tell a story where the table could become an oil rig? And then we could also dive inside the oil rig and into the rooms.

And we could use that as like a dramatic platform, pun intended, for how we could actually frame and structure a story. And so that then just sparked the blue sky ideation phase where it started in that way, but it also felt right. You know, my intuition was that it felt right because I knew I wanted to tell a story about climate. I mean, I think it's just the most existential topic of our times right now in many ways. And I think it's something that connects to younger generations And I think that this technology is also something that we're working in for those younger generations in many ways. So the confluence of the content and the form and then as I started getting into it, just like the world that was emerging in my head naturally, for me it's very intuitive. So when I just feel like it's all starting to click, I just go for it. And that's how it happened with this one.

And then the story kind of rolled out from there. And your background, you come originally from theater because you're also doing film, because I feel like there's a lot of ways in which that you're doing the world building and the spatial design, and that spatial design is perhaps inspiring the type of story you want to tell in the context of that world. And so it seems like you're very driven by the space first, and we can dive into all the innovations you have in this piece in terms of taking what you did with the dial and taking it up to the next level with this kind of mixed reality, multiple layers, and all sorts of stuff we can dive into. But, yeah, just curious to hear about the influences of either theater background or film background all being fused together. Yeah, totally. I mean, I always had a tripartite practice between performance, installation, art galleries and museums, and then film. And for me, film and video always made more sense when I was experiencing it in space. So all of the early video artists that you probably would know, Bill Viola, Mona Hatoum at times,

a lot of different folks who use space and video together were really influential for me. So in college, I started experimenting with making video installations, multi-channel, and then building objects that included video, which was a really interesting challenge. How do you make an object that video is a part of? it's an ephemeral plane it doesn't have depth necessarily so is it a monitor is it you know a mini monitor so I started like deconstructing iPhones and building sculptures using like the pieces of iPhones and iPods actually at first and then of course projection and projection mapping and building worlds so theater was very good to me because the theatrical world came with all the great sort of necessary pieces to do that. Technical directors, lighting designers, scenic designers. And so I worked in groups where we were highly collaborative and structured as though we were an art ensemble, essentially.

My mentors in the performance space all sort of came from the performance group, Wooster Group tradition of collective workshop creation. So get a bunch of people in a room and start building from scratch. You know, we would rarely start, in our case, or in my case when I started making my own pieces, with a script per se. I was less interested in, like, modifying a classic for the purpose of building a contemporary piece. And that's what I did also for over 10 years with the Builders Association. We'd build a lot of work from original content. It might be inspired by an article we read in Wired magazine or something that we saw out in culture where we were like, isn't that curious that this is happening? Like, what is that about? And we would sort of take that as a starting point and then build scripts out of improvised moments and scenes. And sometimes it could start from a piece of video content. Sometimes it could start from an improvisation between a couple of actors.

And so I really learned a lot about building and creating in a room with people and what the value of that is, rather than come up with an idea and then try to impose it on a group of other creators, actors, dancers, etc. Sort of you come up with a seed and then let that seed grow along with all of them and give everyone enough input so that we're all in the same kind of, on the same tune or in the same general space. But then you need to give people a lot of space, you know, so that their talents can flourish. And so it's this constant, like, push-pull as a director in that way where you're trying to give enough space to your collaborators so that they can thrive and yet have enough constant input so that the piece also makes sense and is moving in the right direction for the overall concept. So I think that was pretty influential to start with. But I kind of feel like we all thought of ourselves just as artists. You know, we didn't think of ourselves as making theater per se. or for me, I didn't think of myself as like making video per se.

We did have those areas of expertise, but it was much more collaborative. Yeah, I'd love to hear a bit more around the development of the many layers of experience that you have. You have the table that you have that's like this big, giant, physical table that has like hexagonal LED lights that are lighting up that basically are like a ghost-like reflection shadow of what's being overlaid on top of it through the Apple Vision Pro mixed reality, where you have these 40 Gaussian flat captures that were super high resolution, looked amazing, some of the best I've ever seen of performance capture. But you have it juxtaposed onto this mixed reality pass-through that you see the light from the LED lights, and then you also have LED lights above. You know, I remember from the dial, you had a similar thing where a projection map within augmented reality, but you also had, like, bone conductive, like, you know, audio that also had speakers. And so you also had, like, audio from the Apple Vision Pro, but also speakers from the room.

So there's many layers that you have. And so I'm just curious, as you're developing all of these things together, at what point did you sort of figure out the technological mechanics of how you wanted to tell the story? And just curious to hear about that process as you were developing both the world and all the technical aspects, but also the story as well. So just how that all kind of came together and where you began. Yeah. Well, for me, I think it starts with the story. So I created the story architecture or story shell, like the broad strokes of what the story would be, who the characters are, what the relationships are. I think of my process as a writer-creator in a way like a little bit like Greek mythology. So I kind of write the overarching myth. But then from there, I really like to work with writers, especially screenwriters who are dialogue specialists, who can put the words in the mouths of the different characters in a way that I think...

I work with those people to write the dialogue, but I don't think that's my area of greatest strength. I think I'm a real structure person, so I like to build that myth that architecture and then work with a variety of people to help fill it in And particularly for this story because the protagonist is a young woman who 18 years old I didn't feel that that was necessarily the voice that I was best suited to write for. And I feel like I think it's fair to just acknowledge that and work with others to figure out how to bring those people to life in the most vibrant and respectful way. I want it to ring true. I want it to feel right. And so I actually worked with several different writers over the course of the project. So I started working with Merit Tears, who is a novelist, because I had this theory that a novelist would be a great person to work with in XR because they're accustomed to building not only the dialogue but the world in words.

And so working with Merit was amazing. She is an incredible novelist and also wrote for Orange is the New Black for a couple of seasons and other film and television work. And then she got busy, and so I started working with a friend of hers, Julia Dyer, who helped take what we had at that point and build it out further. And then finally, for the last phase, when we shot it, I worked with a woman, Hannah Pearl Utte, who I had met at Sundance actually in 2019, who I share a few mutual friends with, who I just think is a great writer for, she has a great way of mixing drama and comedy, and I knew that that was the last ingredient that I needed, because I wanted this to feel like, even though it is a drama at its core, that it has a kind of whimsy around this young woman who's living on an oil rig with a bunch of old scientists, and to have some comedy in it, because I just really feel like that helps the audience just engage in a different way,

and engage different emotions within their experience. So the writing team was kind of that next step. And then in terms of visuals, I'm a very visual person, so a lot of the kind of visual concepts are clear to me as they evolve pretty early on. I'm also someone who comes from a background of working with technology and coding my own projects many years ago and so forth, so I understand the technology at a pretty deep level. But I also work with technological teams to flesh that out. I'm not the person who coded this project. Jesse Garrison was the technical director from Night Night Labs. And his business partner, Trey Gilmore, was also important because he is working a lot with LEDs and design. So he and Amber LaRosa, who is a concept artist and designer, sort of helped work with me to flesh out what this table, how it would look in the details. then a good friend of mine Michael Darling who's an incredible technical director and sculptor

was the one who figured out how to engineer this beast of a table that's 14 feet long and nine feet wide and looks like it's floating in space and so he figured out how to laser cut aluminum ribs that would support that massive suspension piece and build the exterior of the table and I told him I wanted it to look like it was, you know, high-tech battleship, basically. And so he found this incredible paint that oxidizes. So you paint a primer layer, and then you paint an activation layer, and then you spray a third layer. And the third layer interacts with the under layer, the second layer, and it actually oxidizes. So it literally rusts. And so you do a scenic paint job on it, sort of, but really that oxidation, and it's totally serendipity. Like, we have no control over what it will do exactly. We have a color palette kind of broadly, but it's that kind of deep rust color, bright

rust color, and it sometimes goes that oxidized green. I just thought it was so beautiful. You know, he brought it up as an idea, and I was like, have you used this before? And he said, no, but I think it's going to be good. So I just went with it. So sort of you start slowly building the pieces of how that will work. But for me, the real storytelling reason is that I feel like I wanted to try to figure out a way to create the closest illusion I could to the Princess Leia effect. These are not holograms, technically. A scientist would not call these holograms. There are no lasers used in the realization of these characters or images. But we want to feel like they are. So the volumetric characters in the headset, for me, when they're not tethered to something physical, they just feel kind of like stickers. They're stuck on. They're amazing and volumetric, but I don't want to stick it on just a flat surface anymore. So that's what we did with the dial, and those were animated characters. So we had the projections to kind of bring the table surface scenically to life,

and then you had the characters. But I still felt like something was kind of missing because they are these beautiful light. they're basically light that's how they feel to me so when we started to get the early results of the gauzian character reconstruction i was even more convinced that i needed something that was light to match with them to make it feel like it was part of a collective whole and i even just today got another question about whether the table was creating had a role in creating the characters that people were seeing above it because the scenery and characters are all, they're created in the headset, but the integration, basically we took a horizontal slice of the game engine world, transposed it down into a flat video file, and that's what's playing on the table. And so you get this connection in real time between the color and intensity, brightness of the splat character or the piece of geometric scenery you're seeing with the video surface of the table. And so...

It's like a diffused shadow of sorts. Yeah, exactly. And we tried different approaches for how to do that the best way, and this is kind of where we are now. Yeah, so it's sort of, I think for some people it will look like it's kind of a shadow, sort of like a downward shadow of the character, and for others they will see it as like the light source is creating the character. And we tried a bunch of different approaches to how to cover the table. So initially we had the idea that we would use diffusion on the table surface first and then perforated metal on top and tried some other different shapes of metal and so forth. And so in the end, we ended up removing the diffusion and just revealing the LED lights below through the perforated metal. That was something that Trey Gilmore suggested in our space. And we tried out a sample of it in L.A. before we came here. And I wasn't totally certain whether it was the right move at first. But then we started to see some of the results in the area of the table we left uncovered,

and I was so intrigued because as you go off axis, it does kind of light up the volume of the octagon. And it's interesting you said hexagon because we tried different shapes. Hexagons are really problematic for tiling, actually. They're beautiful, and we've come to think of them as like the organic equivalent of how to, in terms of design, how to create an organic world, which was definitely what we wanted. But where we landed was this other interesting kind of approach using octagons and pass-through light for the interstitial areas. And I think that ended up actually, because of the way the diffuse light passes through the table, it ended up actually being more organic than the hexagon approach. It was a very interesting and long process of iteration for how exactly the cells on the table would be created. I have to go back and look at it, because I didn't see any gaps. The octagons, you'd usually see holes and gaps, but I didn't realize there's light that's flowing in, so I have to go take a look.

Well, just the way it all fuses together is really magical and awe-inspiring. It's like a technical achievement that I think takes what you're doing story-wise to another level. And so, yeah, I just think it kind of fused it together really well. So the 4D Gaussian Splat capture is, I think, also something I haven't really seen as much as I know Gracia has been doing some videos and work there. But just curious to hear a little bit around the process that you did there, because that seems to be able to kind of open up new vistas of having volumetric performance, where we've had a number of starts and stops in Microsoft Reality Capture, 8i. There's been a number of different entities that have come and gone. Intel had their own performance capture. But I feel like that this Gaussian stats is able to not have the weird, glitchy artifacts you get from the other more photogrammetry base, but having something that looks more fluid and integrated and actually like the most clear volumetric capture I've seen. So it was really quite incredible to see that technological innovation as well. But I'm just curious to hear the process that you use there in order to capture this theatrically staged

performance, you know, all kind of woven together on this oil rig. Totally. I mean, that was the longest part of the technical journey, I would say, is figuring out how to make that in the way we wanted. So initially I knew that we needed to have a pretty large volumetric stage because it's a cast of 11. And I wanted to have sequences where we would have eight or 10 of them in the show simultaneously. So you have two choices. You shoot them individually and then composite them together, which is technically feasible, but in terms of getting connected emotional performances, it just doesn't really work. You can get performances that work from a visual effects perspective for a very short period of time, but in terms of building dramatic arc, performers need to be together in the room to make that work at the level I was looking for, where you're getting deep emotional performances between a father and a daughter, and I needed to stage them in a way that they were moving around. And I knew that this table was essentially theater in the round.

That's what it was with a moving audience. That's how I thought about how to build it. So the techniques that I was using for staging were a combination of cinematic techniques where I knew I was building foreground background in the right way for the audience member to feel like the cinematographic experience they were having was fulfilling visually. But in terms of how they move around the table, I was thinking about it as a classic Shakespeare play. I mean, this was, you know, the Globe Theater and figuring out how do you stage in the round for an audience of 10 moving participants and also get that cinematic feel. So I knew I needed the large volume to start with. And then the early days, as you mentioned, especially with mesh and texture, it was always a problem of the crinkling and the missing pieces underneath the arms and the legs and the back and like any fabric. You had really mixed results with fabric.

So I was watching the development of Nerfs and Gaussians over the last five years or so and studying what was possible, talking to friends who were making them, and getting a sense of, like, if I'm targeting my arrow at a moving target somewhere down the road, what might I expect for when these things start to become feasible? And also, what devices might I be able to play them back on? because that's the biggest question is you can make a beautiful volumetric capture for point cloud and render it for visual effects for film and television. It looks fantastic, but you can't put that in a real-time headset. So you need something that can be compressed and still look good. So this is where the technical side comes in, and this is where I feel fortunate that in the early part of my career I was my own technologist. So I'm not the greatest technologist in every area now, But I know a lot about it and enough to know how to approach it as a director to pick what I need to use to make it happen. So the first stop after doing the volumetric, well, this show was supposed to premiere in 2020 at Tribeca.

So it was shot in December of 2019. It was supposed to be at Tribeca in 2020. Lauren Hammonds had programmed it after seeing the dial. And we all know what happened in the spring of 2020. And so we packed it away, and then I was waiting for the moment to bring it back. So you had already shot it by then? Yeah, we shot it in December of 2019. Oh, wow. Which tells you a lot about what the importance of high-resolution source and then a really competent, intelligent reconstruction model can do. So I first started working with Infinite Realities out of the UK, and they were great partners. Their Gaussian splat results were the best I'd ever seen two years ago. We started working with them, and where we arrived with that experiment was some incredible results, very beautiful imagery,

but we were struggling because we didn't have a compression paradigm and we didn't have a methodology for playback. like for a 23-minute experience. So we all were very candid about that with each other. Their platform was built for shorter video memories, these sort of shorter snippets, and they were developing a workflow to build out to longer time frames, but it wasn't there yet. And so they actually introduced me to Xiaming Sun and his white paper from last summer that led to him creating 4DV AI. And there's no gen AI, there's no AI of that kind in the show whatsoever, but there is a machine learning model that he's created for reconstruction and training of the Gaussian Splats. And he had a pathway to compression, and the technology that he created and the partnership that he gave to us, to me, was really incredible because the first conversation we had

he immediately said so are you basically building the first interactive film and i said well there's examples of interactive film before but in terms of building something that i think will look photorealistic and have a diverse and recognizable cast you will be familiar with and have a story that feels connected with arc over the 23 minutes i think the answer is yes and so he said i think we should do this together and so obviously that technology is quite expensive and so he sort of became a partner in the project and they had to do the reconstruction tests and their system they shoot differently than we shot it initially so they had to do a lot of modifications to the software and the workflow to make it work but once they finally cracked it the results were undeniable so what you saw when we started to see those results I knew this was going to work because they had the reconstruction, the quality, the compression, and the pipeline

suddenly clicked into place. And so that really was in January of this year, maybe November, December, January of this past year, so less than a year ago, that started to click. And then we had to do the processing through the spring. And so I think we finally had all of the footage processed maybe by May or thereabouts. So it was a lot of iterative testing and going back and forth with them, figuring out what the color pipeline was and all of that sort of thing. And they were great, I mean, because the quality, I think, is undeniable. But they also had to kind of, like, reverse engineer a process for this larger capture volume. And the way that they're thinking about doing larger staged work like this in the future is quite a bit different now, six years later, than we shot it originally. So I think that it was a great partnership because I think this really proves out what that larger dramatic cinematic staged volume can feel like.

Yeah, and were you using the Intel? I know they went out of business and they no longer exist now, so I don't know if you can say anything about that or not. I can't say anything about where this piece was shot. but if you care to speculate, you are welcome to speculate to your heart's content, but I can neither confirm nor deny from a legal perspective where the footage came from. Okay well Intel been the only large performance capture that existed at the time that you shot it and there been a few pieces like Running from Kira Benzing was another piece that was shot on that stage And maybe there's some other footage that is out there. Okay, well, so I wanted to ask about the Apple Vision Pro because I feel like that's another part. Apple, I don't think, was necessarily thinking about location-based entertainment. It's not really meant for LBE in a lot of ways. but they do have, like, the best screen, and the experience looks amazing, and so I'm just curious to hear about the process of compositing from Apple Vision Pro with all this physical installation.

Yeah, I mean, I think what you said is exactly right. I was looking to try to do this in glasses because it would feel lightweight. It would feel more of a social experience in a way. You're not boxed in. You're not dealing with camera pass-through. but I tested all of the glasses that are out there and they're just not ready yet. And I don't fight the feeling. I want the work that I make to deliver on the promise of the medium. That is the most important thing to me. It's not about whether or not it's in glasses or headgear or headset. It's about do I deliver to my audience the promise of the medium. And the promise of the medium is immersion and connection to characters, participation in a story, engagement in a story. Those are the things that I'm focused on. And so I think there's some amazing glasses out there, many of which I now own in my cemetery of technology, and many that I've had companies that I've worked with to try to develop different things.

We worked carefully with one company to develop edge compute capability where they didn't have it before, a variety of different things like that. And I think there are applications for glasses right now that are acceptable for what they're capable of processing, displaying. But for what we were trying to do, the level of detail we needed and the processing power we needed, there's no one that's even anywhere close to that in a standalone form factor. So we looked at doing edge compute or something where the glasses would receive the data remotely. but still for me the resolution and contrast ratio was not high enough to deliver the delicate detail we had i mean when you like you said when you look at our main character lani in the first scene she's waking up and she's moving around and her long curly hair is falling and moving around and you're seeing every strand of hair you see with incredible detail this kind of early morning moment And so I didn't want to lose any of that. And I think that's what the Vision Pro was able to display.

And so I think as Apple has started to expand the ecosystem a little bit and they've seen the project now, I think that there may be, I hope that there's some interest in how this could be a part of that bigger picture. I think there's a way that the immersive film aspect can live together with something that would take advantage of the six-doff capabilities of the headset, that I don't think they're mutually exclusive. I think it makes sense that a distribution model based on immersive video is a great place to start trying to, like, distribute and monetize. But I also think that for the promise of the medium, to go back to that phrase that I think is, for me, my North Star, I think the promise of the medium when you put on a headset is to be in an immersive world that you have total freedom to move through. Not just to look at, but to actually move through and be immersed in.

So it has to be volumetric, three-dimensional. It has to be in some way participatory because that's how we think about the world that we live in. And so I think that these experiences are just the logical extension of some of those early content approaches. And I think the headset is built in a way that it's ideal for doing both. And so I think, I hope that in time those conversations will start to mature. But we had some really great late conversations with the Apple team. in particular a good friend Jessica Brillhart who came on in the last few months so she and I have known each other for a long time and have had many many many conversations about immersive experience and VR, XR, AR all of it and so you know and Elliot Graves also who's part of that team and many others I had a great conversation with Serenity Caldwell as well and so I think all of those team

members are very interested in what we're doing. And so I'm hoping that bringing it to life in Venice and then in January, we'll also be at Lincoln Center for two weeks in New York. Between these two venues, I'm hoping that all of them get to see it and sort of, I mean, the biggest responses I've gotten here is just people are saying that they haven't seen anything like this before. And so, and I understand that. It's not something I can describe or show in a deck or a flat video capture. You have to see, you have to experience it. You have to be immersed in it. So I'm very optimistic that when folks from Apple, for example, or any other technology developer come and see the approaches I'm taking as a storyteller and as an artist, I think they're going to see that this really is next gen storytelling and that this is not niche. This is not a niche format that I might use LBE as the starting point, but I've had great conversations with the Apple developer relations team about like, they're like, you're ready to make this

into an at home, like we're there for you. And I think that's the right, I appreciate that. That's the right approach. And I think, so I think the ecosystem will grow around those things. And it's, it's really about how do we create these connected stories? And like, I always hesitate to think about the killer app concept because I kind of feel like it's going to be about a lot of these sort of experiences that start to really onboard the community into experiencing narratives in their home. It's different. And so we need to give it time to gestate and mature. And so I was very thoughtful about this experience being something that has an onboarding paradigm that welcomes people into it. I think the analogy I use for my students at CalArts about how to think about immersive storytelling is that you can think about it like a dinner party. So a dinner party is not just one thing. There's drinks over here. There's a couch to sit down and rest over there.

There's snacks and nibbles sort of distributed throughout. Maybe you want to go smoke on the patio, but all of those different areas are part of the environment of the party that are essential to keep things moving and to keep people comfortable. And I think of them as offerings. So these various offerings around the space are how you welcome someone into a story, just like welcoming them into your home. It's not about forcing or coercing them, you know, but some things like maybe food, for example, maybe some snacks need to be distributed in multiple areas around the house, you know, and that's true of story as well. Some key story elements need to be brought up or revealed maybe more than once, maybe in different areas of the story if you're thinking spatially. So I think it's just as a creator, those are just new paradigms, you know, that we need to consider, like what are the fundamentals of writing or creating for these new formats? Yeah, and, you know, there's been a phrase of tabletop scale, and I think this is like a big table, a 14-foot by 9-foot table.

So the scale that you have is much larger than what most people would have at home, and, you know, you need a very large space. So imagine, you know, you're already playing with scale a little bit. So I guess theoretically you could shrink it down a little bit to be like someone's room, But I think it would maybe lose a little bit of the other aspects of the physical installation, which is the other lighting effects that I think are hard to completely recreate because you are a layered creator that has things at the near scale, at the mid scale, and the far scale, and even the audio. You have, like, I don't know how many channels of audio, but I heard, like, soundtrack happening in the background, but I hear all the dialogue coming through the Apple Vision Pro. And so just curious to hear if you want to say anything about the audio design in terms of the layered audio from both the physical audio speakers in the room along with the dialogue that's coming through the headset. Yeah, totally. I mean, I'll just address the first point. I think you're absolutely right about the full location-based experience being the richest, the most layered, the deepest experience.

But I kind of think of that in a way akin to how we're ingesting various media now anyway, that the cinema is the location-based experience of the film. When you watch a movie on your phone on an airplane, it's not the richest experience, but I'm not going to deny that experience to someone who otherwise would just not experience it at all. So I think the concept for me, if I were to release this at home, is that you'll get the whole virtual table with all of the elements on top, and then you'll be able to scale it however big your space is. If you can cut the rug and move the sofa and give yourself 14 feet of space, you can have a 14-foot table in your house. But if you only have a 7-foot table and everything's half the size, it would be here in the location-based experience. I'm okay with that because I want this to reach as many people as possible. Part two of your question, the audio design was very intentional. I want to give credit to Jesse, the technical director, for figuring a lot of how that works out.

Also, my two longtime collaborators, Sage Lewis, who's the composer, who I think is incredible for figuring out how to use music in this spatialized way. Because they're all, and Scott Hirsch, the sound designer, they both work at the highest level as film and TV composers and sound designers. so they bring expertise from that world into this world but they're also so curious about this world so they've both learned how to use unity so they both program their own sound sources now in my projects and so i love working with them because they are bringing that sort of classic media production value that is at the top level but they're bringing it to this world and using the tools for immersive storytelling in unity, which is our game engine platform to do all of that spatialization and layering. And we then work in to go back to the beginning of the conversation about workshop practice. We all work together in a finishing workshop for the last month.

In this case, we did it in Glendale and it's very much like finishing a performance work. I find it's all of us there in the room together for an hour or two. We're doing our independent notes and then we bring it together and test some pieces, and then we do a run and make some adjustments. And so it's a very collaborative process. And Sage and Scott have also got really, they've become very attuned to how to use each of their strengths to the other's advantage. So the finishing part of it is like very intertwined in terms of like, I mean, how do you do a mix for eight headsets at a time with quadraphonic audio around? There is no paradigm for how to do that, you know? So we have to invent that paradigm, and each project is a little different. We learned a lot from the dial, because my premise with the bone-vibrating headphones in the dial was that the audio from the headphones that did the spatialized dialogue and sound effects would blend with the quadraphonic music around you, the score.

And indeed it did. Your brain puts them together like one thing. And for this project, we use the same paradigm, but with the Vision Pro creating that spatialized dialogue and sound effects layer. But my other supposition was that we could also add some music or like music-like cues into the headset and vice versa. So we do some big sound effects in the speakers, which is normally where the music lives, and vice versa, some music augmentation in the headset. in fact to the point where Liz and Michelle from Venice Film Festival came and demoed the project or experienced the project two days before the festival opened and when they left they said we have to talk about sound and I said okay what do we need to talk about and they said well some of the sound that we're hearing like the music seems like it's way too loud it'll maybe disturb other pieces around you And I said, well, actually, I think we should go out in the hallway and listen to it because I built it with the idea that we augment some of those bigger moments in your headset.

So you think it's coming from the speakers. And some of it is. That's how I'm doing the quadraphonic spatialization. But I'm augmenting it with music in the Vision Pro speakers at that moment because I know in a festival context that I can't drown out other pieces in the gallery. And so we indeed did step outside, and they said, this is unbelievable. We had no idea that so much of it was coming from in the headset. And so then we checked with our neighbors for when their quiet moments in their pieces were, and we adjusted a few things accordingly, but ultimately we have the piece that we all wanted. But I think that's like another thing, is if you're thinking about showing work in a location and in space, you also have to be cognizant of what that means. it's not a movie theater where you get total free reign over everything all the time when we take this to lincoln center we will have a full room that we can do what we want but in other cases i want to be prepared to show at a fantastic festival like venice and i'm going to be i've been doing this long enough that i want to be mindful of how to exploit the space and be a good neighbor to the

pieces that are around me you know so and we can do that it's so that's what some of what this layered sound and other things kind of those opportunities give us that's really amazing yeah i didn't notice that the audio was coming through the headset i think i just it was all blended together in my mind anyway so i wasn't able to discern the difference and you shouldn't be able to because if you can tell the difference then then it becomes a technical thing that i don't that i don't want you know so it's like it is about how the magic trick unfolds yeah no no i was hearing about it i'm a little bit awestruck that you're able to pull that off. That's really amazing. I did want to ask around this immersive theater-esque mechanic of moving around this 14-foot by 9-foot table to see sometimes there's multiple perspectives of scenes or two scenes that are talking to each other or maybe even one part where you actually have three or four scenes that are unfolding that you can walk around and see everything around. And so it reminded me of something like Sleep No More. It's the closest thing I've seen in a VR piece

that allowed me to give this walking through space to get access to different narratives within a VR piece. But I thought it worked really well. It's a great mechanic, but just curious to hear some of your inspirations in pulling that part off. Yeah, absolutely. I mean, I think, again, we go back to the promise of the medium. I feel like I knew for this piece that I wanted to have some moments that had branching narrative. So we start with a scene at the beginning of the show where there's a dialogue that emerges between a character in one room with a character in another room using this kind of virtual hologram communication device that we use as a mechanic, you could say, in the show to communicate between rooms on the oil rig because these oil rigs are gigantic. So we wanted to kind of make some sort of holographic technology a core element. And, of course, you see the table that you're experiencing the project on diegetically in miniature in the show itself. So you see the characters at a table that is exactly like the one you're standing at.

So it's a little bit of like an Ouroboros as far as how you embed that into the storytelling. But the idea that you start with two scenes where the two characters are having a continuous dialogue is a very soft onboarding methodology. So you can hear both sides of the dialogue, and when you move to the other side of the table, you hear it mediated through a sound effect that makes it sound like it's coming through that hologram projector system and vice versa. So that a really soft open to branching narrative because you literally can miss anything It a dialogue just between the two rooms So then two scenes later when we split the show up into four rooms you already kind of know oh there you see the other rooms sort of appear and then the walls between them go up and you know they're there. And so people are flocking to the next room. And so you've got four rooms you can experience. So then the question is, how do you write for four different room experiences happening simultaneously? And so So for me, it was that moment is about each of them is the same central question, which is what do we do about X?

And so in one case, it's do we have enough X? So do we have enough food? Do we have enough medicine? Do we have enough weapons if they turn violent? This is about the core narrative driver of the whole show is at the beginning of the story, Lonnie, our young female protagonist, he's a boat on the horizon. and it's the first boat or sign of civilization our group of scientists who are on this oil rig have seen in the five years that they've been stuck out there since this massive tsunami wipes out the coast. So their summer sabbatical becomes a permanent habitat, and so they're out there to build a sustainable city at sea, and this tsunami suddenly turns that hypothetical into a reality. So they're stuck out there. Five years later, they've built systems and managed to save their solar panels and put together a broken generator a hundred times. And so we experienced that central question of what do we do about this group of immigrants on the boat through the lens of those different resources.

so do we have enough food and you hear a conversation about whether their cricket farmer bugs thinks there's enough insect protein on the ship to support an influx of new people and then you move around to the next room and you hear our doctor Gita talking about well do we have enough antibiotics do we have enough bandages do we have enough medicine if they show up injured or diseased what would we do about that And so the arc of those four rooms, if you think of them as horizontal layers and vertical, as you move around, I wanted it to flow from one topic to the next. So they all basically have the same arc in terms of how you would structure the storytelling. And they're all based on a similar central question. It's just a different topic. So that was the hope, is that you moved between those different rooms and you sort of feel like the conversation flows, even if you started hearing about food and now you've moved into medicine, it flows.

And then also when you flow out at the end, I think you don't have any nagging questions about whether you missed something because you kind of know what the flow of the overall arc and question is. So that's how I thought about it structurally. Yeah, I thought it worked really quite well because I was able to walk around fairly quickly and get a sense of what each of the rooms were talking about. Then they were all talking about the same questions and topics. So it was kind of interesting to see the different characters, some of them being introduced sort of for the first time, and then you are hearing more about their perspectives. And so I did want to talk around the ending of the piece. And I guess for anybody who hasn't seen it and wants to see it, stop listening now because we're not getting to the spoiler area. But I felt really emotionally impacted by the piece because there's a lot of climate change type of stories that end up being very pedagogical or talk about the facts. And I feel like you're able to address the core issue around what do we need to do in terms of to solve the problem, which is essentially be in relationship to each other and collaborate and be open.

and I feel like you managed to create a narrative arc around this core central question that had a lot of dramatic tension, that I feel like had an incredible payoff with how it ends up, but essentially the archetypal use of the piece speaking out of, hey, all your elders, you kind of screwed everything up with not taking action. You're doing the same thing. We need to think about this in a new way. And it was really, really quite powerful. So, yeah, I'd love to hear any reflections on that. Yeah. Thank you, first of all. That was one of my ultimate goals for the piece, was getting to that place where the ending feels really emotionally connected and powerful. So thank you for that. I appreciate that feedback. I mean, I think I thought about a lot of different types of endings. So the structure of the piece in general is that at the end, our young female protagonist, Lonnie, sort of goes through a series of reminders about

who her mother was who founded this expedition. Her father comes and gives her her grandfather's rifle to protect her, and she rejects it and throws it in the ocean and says goodbye to the past. And then she rewatches her mother's final video recording before her death some years prior due to there being no more antibiotics left on the rig. and her mother talks to her about what her responsibility for the future is and how she needs to trust her instincts and how she needs to ask for help when needed but also become the adult woman that her mother knows she can become. And so there's this coming-of-age element to the story that I think is central structurally. There's also a mother-daughter and father-daughter story at play, and there's a nice tension between those two, that there's like a fully formed family relationship, but the tension between what each of those people means to Lonnie, their daughter, I think was helpful for me in terms of structuring that dramatic arc for the final moment.

But then the question was, okay, if she steps into her role as a woman and as a leader, which was the big thing, was how do we sort of create this paradigm for young women to step into leadership roles? She steps into the role as a leader with all of these older scientists on this oil rig. So what does she do? How does she do it? So she asks for a vote. She says, this isn't just your decision, Dad. This is all of our decision, and I think we should vote. So they decide to take a vote. So how do you position the audience within the structure of a vote that feels really meaningful? So, I mean, I think the knee-jerk impulse is to give the audience a vote. But personally, I didn't like that idea. I thought it's going to feel just super didactic. Like, it's going to feel like I hate as an audience member being asked to interact in a way where I feel like I'm just, like, a functionary. Like, it's sort of...

There's no real choice. there's no real choice or like it feels like if you make a choice like how do you find payoff and i was just like you know what sometimes the idea of interaction being about button pushing or binary choice making isn't really what the world feels like most of the time for me the better way to think about it is how do we participate in this in a meaningful way and so for me i thought okay so number one i want to feel like these members of the boat of the oil rig community are now suddenly around me in a new way. I want to feel like I'm a part of them in a different way. So we have a special moment at that point that plays with scale that makes us feel like we're part of the full oil rig community. So I'm not going to spoil it totally, but that's step one. Then step two is, so then we feel like we are part of that group, but how do we think about the framing of the final choice?

And for me, that becomes about how the group members vote, and the final vote, as we might imagine, is cast by Lonnie's father, who is the person who is the most incentivized to be conservative, protective, and ultimately not take risks that could put his team in peril. So your audience can imagine what he might have voted. that he votes in favor of his daughter against possibly his better judgment, against his personal desire for protection for the rig community, which I thought was really an interesting way to problematize it. And then the final gesture that we experience is when we see Lonnie as kind of a large hologram above the oil rig, which is like a concept for how they can communicate outward to this approaching boat. and when she looks down at us we have yet another flip or reversal in our perspective about what our

point of view is so again i won't spoil it but we have this shift in perspective about whether we are actually a member of the oil rig community the scientists we've been empathizing with throughout the story thus far, or whether we are actually empathizing with this group of people who are approaching on the boat, the immigrants or the others. And so that was what I found to be the more interesting approach to how we immerse the audience in the story is I can shift your perspective. I don't actually need you to raise your hand and vote. And that felt to me like the more powerful and emotionally connected way to do it. Yeah, I'm glad you went that way because I was wondering if I was going to have to vote. I'm glad that you made the choice that you did. So yeah, it was super, super powerful. And even as you're recounting it, I just feel these emotions well up because I feel like it's such an important and vital story that I hope that gets out there and spreads in a way that people really take it in and think about what it means

to be in relationship to the world around us and to other people. So yeah, I guess as we start to wrap up, I'd love to hear what you think the ultimate potential of immersive storytelling might be and what it might be able to enable. Oh, my God. I think the sky's the limit. I mean, I think we're living in a time where, you know, younger generations are living in these atomized screen-based worlds where they're hungry for connection. And I think I have no problem with using screens, vision pros, glasses, any of these technologies as communication tools to bring us together. But I think ultimately the live human experience is what everybody's craving. And that's true from the younger generations to the older generations. And when we look around at immersive performance as one example, or even like immersive installation, these are things that are runaway hits with the younger generations. When I show the work that I've made in headsets to kids who are 12, 13, 17, you know, even in their early 20s, they love this. And they're just trying to, they don't know how do they step into these creator roles. They want to be audience members. They

want to be creators. And so I think the sky's the limit for the immersive storytelling world because I think participatory storytelling is truly native to the younger generations. They're accustomed to participating with their media. They're accustomed to participating digitally with social platforms online. And I think creating healthy, positive, influential, meaningful stories that they can participate in in these ways that feel fun and like where there's comedy in the stories too you know like I really worked hard to like make the moments that are fun you know Bugs throws a Beatles concert for his crickets one night or Bill and Bugs are hypothesizing that there could be a goat on board the approaching boat and they might be able to make pizza you know with the cheese they could make you know finding these moments that are just fun that still are part of that world, but like we're not dour all the time, even in the face of a really dark political landscape or a turbulent climate.

Like we're still going to find moments of just like small personal connection. And I think like highlighting those things is really helpful and building these worlds that feel more native to younger generations is going to communicate this stuff to them because they're the ones like Lonnie who are going to have to step into their adulthood and step into their leadership capabilities to turn things around. Because those older generations, that's us. You know, in the story, it's the parents and the older scientists, but it's us. We're the ones who are kind of inactive about these topics. Like, climate inaction is the problem. The data's clear. You know, the solutions are also, in some cases, pretty clear. It's just the inaction socially that is the biggest impediment right now to solving some of these problems. And so I just want to do my part in some small way by making this little story that hopefully will spur a little bit more action in that area and also is sort of creating some entertaining platform or paradigm for how to tell stories to the younger generations that are looking for this something new.

Right. Is there anything else that's left inside you'd like to say to the broader immersive community? um i mean i think i'm excited for how we continue to expand these formats and think more deeply about what some of the storytelling i hesitate to say rules but some of the kind of storytelling approaches and paradigms we can build together i think we can learn so much from each other and so building ideas for how to write immersive scripts and like build these immersive projects and ultimately, hopefully, start to create spaces for this type of development. There's some that are out there already that I've been a part of, and I think kind of finding ways to create opportunities for all of us to keep building collectively and productively at a time, like you said before, where funding and opportunities are unfortunately lower than we would like. I think that we would all love it if there were more opportunities for this type of creation

and more funding to build these projects. So hopefully, like, by working together and building up this ecosystem, incorporating more forms, so we're pulling in installation and art, we're pulling in film and storytelling, we're pulling in theater, we're pulling in gaming, you know, sort of building these worlds that are more integrated, and then hopefully, ultimately, as a result, feel more like this kind of Gesamtkunstwerk, to steal a phrase, this sort of like unique whole art form that we're moving towards in the immersive world. I think the promise of immersion, particularly in storytelling and art making, it's just so big. It's so rich. It's so huge. It's like the tent is giant. There's so much room for all kinds of different creators and projects. And so I feel like this community in general is a very optimistic one. And I think that serves us really well as a group. And I love being a part of this.

Awesome. Well, Peter, thanks so much for joining me. I really, really enjoyed the Empire at Sea. And thank you. I have so many technological innovations that create this really awe-inspiring experience that has so many technological innovations, but also the story at the heart of it is able to really have this intense, dramatic moment with a lot of payoff. And I think a message that I can really get behind and really moves me to think about it. So, yeah, just a really amazing job. And, yeah, thanks so much for joining me on the podcast to help. share a little bit more around your journey and process of making it. So thank you. Thanks for having me, Kent. It's a pleasure to talk. Thanks so much for listening to the Voices of VR podcast. And I just want to say that I'm really trying to live into the transformation economy, meaning I want to give away all this information for free, make it public accessible for not only you, but the entire community. And if you find value in that, then please do consider supporting me on Patreon. You know, I'm on this journey of trying to reach my 1,000 true fans. And if I have just $5 a month from that 1,000, I'll be in a much more solid and stable financial position. So really need your help to be able to

get to that point. So if you enjoy what I'm doing here at the Voices of VR podcast, then please do consider supporting me at patreon.com slash Voices of VR. Thanks for listening. Thank you.