
![#1791: [2019] 「The Dial」の技術的成果](http://voicesofvr.com/wp-content/uploads/2025/04/Voices-of-VR-Podcast.jpg)
#1791: [2019] 「The Dial」の技術的成果
Voices of VR
要約
Voices of VR が、2019年のサンダンス New Frontier で展示された AR 作品「The Dial」の技術ディレクター Jesse Garrison とソフトウェア開発ディレクター Brian Chasalow に取材した回。プロジェクションマッピングを施した卓上模型と、iPhone の ARKit によるAR重ね合わせを組み合わせる際のトラッキング、遮蔽、音響の課題と工夫が語られる。AR の将来、5G、ソーシャル体験への期待にも話が及ぶ。
この回のポイント
- ●動的に変化する投影映像の下では ARKit の通常のトラッキングが機能しにくく、家の各面の画像(影なし)をターゲットとし、ワールドトラッキングと併用するハイブリッド方式にしたと説明している。
- ●遮蔽用の不可視メッシュは画像に素早くスナップさせ、他のコンテンツはゆっくり補間して移動させることで、精度と体験の違和感のなさを両立した。
- ●ズレが大きい場合や、電話が10フィート以上離れるなど明らかに誤った値の場合はトラッキングをリセットし、前後の面など信頼できるターゲットを優先する制御を入れた。
- ●骨伝導ヘッドホンは扱いやすい一方、Bluetooth の遅延(約0.5秒)への対応や EQ、有線版アプリの用意が必要で、4chサウンドも機材の相性に苦労したと述べている。
- ●展示環境ではハードウェアと空間の寸法が既知のため最適化しやすく、家庭向けでは端末が不明で難しいという対比が語られた。
- ●今後はAR クラウド(6D.ai など)、世界規模トラッキング、動的な遮蔽、ソーシャルAR、5G による大容量データ配信に期待を示した。
章立て
制作陣の紹介と技術概要
Jesse と Brian が自己紹介し、プロジェクションマッピング、AR、音響の座標とタイムラインの同期という全体像を説明する。
動的な環境でのトラッキング課題
通常のAR は静的な対象を前提とするが、本作では光や影が常に変化する。家の側面画像をターゲットにする工夫や、ワールドトラッキングとの併用を語る。
制作者の経歴と音響の工夫
演劇や VR 分野での経歴、WWDC デモでの Apple との協業を紹介。骨伝導ヘッドホンの遅延、EQ、4chサウンドの課題も話す。
自己補正とフェイルセーフ
遮蔽メッシュは即座に補正し、コンテンツは緩やかに補間する設計や、ズレが大きい場合のリセット、信頼できる面の選別を説明する。
白い模型の難しさと応用の可能性
投影なしの白い家なら追跡はほぼ不可能で、深度センサーが必要になると話す。大規模ステージなどへの応用や社会的な体験の可能性にも触れる。
来場者の動きとQA
来場者の行動を想定したエッジケース対応やプロローグの役割、互いの会話を促す設計について語る。
未解決課題とARの未来
動的トラッキングや遮蔽、ARクラウド、文脈認識、端末選定(iPhone XS Max)やバッテリー運用を議論する。
究極の可能性と5G
AR/VR の融合、ソーシャルで体積的な未来、5G による大容量データ配信への期待を話して締めくくる。
解説記事
2019年のサンダンス映画祭 New Frontier で展示された AR 作品「The Dial」。その技術面を担った Jesse Garrison(技術ディレクター)と Brian Chasalow(ソフトウェア開発ディレクター)に、Voices of VR のホストが話を聞いた回である。なお、この回は後にPeter Flaherty の新作「Empire at Sea」でも Jesse が開発リードを務めたことを受けて、改めて公開されたと冒頭で紹介されている。
投影と AR を重ねる難しさ
The Dial は、卓上の白い家の模型にプロジェクションマッピングで映像を投影し、その上に iPhone を通して AR コンテンツを重ねる作品だ。二人は、投影と AR が世界座標や実寸で連携しなければならず、さらに投影・AR・空間オーディオ(ヘッドホン)・部屋の4chサウンドのタイムラインも同期させる必要があったと説明する。
一般的なスマホ ARの追跡は、静的な世界や画像・物体ターゲットを前提とする。しかしこの作品では光も影も常に動いており、ARKit が頼れる不変の「正解」がない。そこで二人は、影を除いた状態で家の各側面の画像を撮り、それをターゲットとして使った。さらに部屋全体のスキャンによるワールドトラッキングを併用し、ターゲットが見えないときの保険にしている。3Dモデル追跡や画像追跡も試し、ハードウェアによる解決も検討したが、それは研究プロジェクトに近いと判断したという。3Dモデルは遮蔽(模型の後ろにある木などを隠す処理)のためにのみ使われ、位置合わせには画像を使う。
速い補正と遅い補正の使い分け
特に興味深いのは、補正の設計である。画像ターゲットは最も正確な基準なので、不可視の遮蔽メッシュは可能な限り素早く画像に合わせる。数センチずれて木が家の裏から見えてしまうのを防ぐためだ。一方、地面などの他のコンテンツは急に動くと鑑賞の妨げになるため、ゆっくり補間して目標位置へ近づける。
さらに、大きくずれた場合はトラッキングをリセットして画像ターゲットの角にアンカーを移す。展示空間の条件が既知なので、たとえば電話が10フィート以上離れていると検出されたら誤りだと判断できる。ARKit からのデータを信頼するかどうかを判断するロジックも組み込み、大きな前面と背面を最も信頼できるターゲットとして使い、小さく回転に弱い側面は慎重に扱った。人の影でターゲットが見えなくなっても、ワールドマップが最後の位置を保持して再検出を待つ。
音響とデバイス選定
音声は骨伝導ヘッドホン(Aftershokz の Trekz Air)で届けた。周囲の音を遮らないため、参加者同士が会話しやすいという狙いがある。扱いは容易だったものの、専用の EQ 調整が必要で、世代的なBluetooth の遅延(発言では約0.5秒)に合わせてタイムライン全体をずらす処理を入れた。会場が騒がしい場合に備え、有線の通常ヘッドホン向けのアプリ版も用意したという。4chサウンドは、DAW 向け機材が Unity や Windows と相性が悪く、結局コンピュータ内蔵出力を使ったと述べている。
端末は iPhone XS Max を採用した。ワールドトラッキングは負荷が高く、アセットも多いため、当時最速の機種が必要だったという。iPad では大きすぎ、画面が大きめの XS Max が中間的な選択だと説明する。10台を用意して充電をローテーションしたが、バッテリー面の問題は想定より少なかったとしている。
体験設計と将来への見方
QA では、来場者が卓を走って回るなどの想定外の動きに備え、シーン間の遷移を工夫した。冒頭のプロローグは、観客が腕を振って仕組みを試せる時間として設けた。実際の展示ではほとんどの人が卓を反時計回りにゆっくり回ったが、ロサンゼルスのテストではより実験的な動きが多かったという。
将来については、世界規模のトラッキングを担う AR クラウド(6D.ai を例に挙げる)、動的な遮蔽、人や動く物体を静止物から分離する意味理解などが課題として挙げられた。二人とも、消費者向け AR ヘッドセットが普及し、社会的・体積的(ボリュメトリック)でワイヤレスな体験が増えると見ている。5G は大容量データの配信や、クラウドベースのトラッキング基盤に役立つとの見方を示した。
まとめ
この回の核心は、「条件が固定された展示だからこそ使える現実的な工夫」を積み重ねて、動的な環境でも安定した AR を実現した点にある。画像ターゲットとワールドトラッキングの併用、補正速度の使い分け、異常値の棄却は、現在の空間コンピューティング開発にも通じる設計思想だ。一方で、家庭向けではデバイスも環境も不明という制約があり、二人はそこを今後の課題として語っている。2019年時点の発言なので技術状況は変わっているが、AR 体験の設計を考える上で参考になる具体例が多い。
文字起こし(英語・自動生成)
The Voices of VR Podcast. So they were both working on a piece called The Dial, which was sort of like a augmented reality piece that was showing at Sundance back in 2019. And so there's lots of different things technically that they're doing, many different layers of audio with having like a projection map onto a small tabletop model, but then also doing augmented reality overlays on top of that and just a really well composed and constructed piece. I did an interview with Peter Flaherty that has already aired. And so Peter was back this year with Empire at Sea, and Jesse was also back as sort of like the lead developer on that project.
And so I just wanted to air this episode just to reference the work of Jesse and the amazing work that he's done both on The Dial as well as on Empire at Sea. So we're covering all that and more on today's episode of the Voices of VR podcast. So this interview with Jesse and Brian happened on Tuesday, January 29th, 2019, at the Sundance New Frontier at Park City, Utah. So with that, let's go ahead and dive right in. My name is Jesse Garrison. I'm the technical director of The Dial. My name is Brian Chassolo, and I came on as the software development director of The Dial. Great. So I've had a chance to run through The Dial a couple of times, both as the navigator and also as a passenger. And so maybe just give a brief overview of the context of what you're working with and how you think of this project. Sure. You got most of this from Peter, right? Yeah. I just want to hear the technical side from your point of view. Yeah. Okay. Well, yeah, it's a lot of moving parts. There's obviously projection mapping and augmented reality that are having to work together and communicate in terms of coordinates in world space, actual dimensions, in-game dimensions, that sort of thing,
as well as dealing with synchronizing the timelines of the projected content, the augmented content, and the sound with the spatialized headphones as well as the quadrophonic sound in the room. So how was the QA process in this piece? It was difficult, but we were queuing as we were developing, just as many people as could come through and keep trying it, especially on our last week when things started to come together. We did a round of QA and brought in friends and family to test it out. But it wasn't until the last couple days that we really dramatically changed some of the tracking algorithms and just the procedures for how that relocalizes. And that's when it started to really come together. We were really nervous that it wasn't going to get there, but it did at the end. I'm wondering if you could describe to me a normal use case for most phone-based AR and all the, I guess, non-normal use cases that you're doing here with this projection map
and maybe some of the specific details in terms of how it usually works in terms of tracking and then what you had to do in order to make it work with all of the ambient lighting and everything else, projection maps and everything, all the stuff that you're throwing into the mix here. Yeah, well, I mean, most AR tracking solutions are anticipating static things to lock onto, either a static world, build a world map, or some sort of target, like an object target or an image target, and those are looking for specific points of contrast and et cetera. So in our situation, everything is moving. Basically, everything is changing. Everything that it would look to as a constant is dynamic for us. There's no real ground truth of what the correct image is for ARKit to track against because we have dynamic lighting with normal maps on each side of the house. So as you walk around and it's trying to detect, can it find that side of the house?
because it's using the sides of the house as the ground truth. You can't really use the ground plane because there's too much, there's not really a unique enough image target, and the ground plane imagery changes constantly. So we decided let's use the thing closest to a ground truth, which is the images of the sides of the house. and so we ended up writing some software that captures each side of the house with lighting but without shadows because in the normal experience it has shadows but those shadows are going to change and ARKit, the best you can give it is a solid set of unique features and so what we did to account for that was trying not to rely
or really trying to ignore the fact that shadows are going to be projected on top of this house and there's really nothing we could do about it. And we tried a couple different techniques. We tried 3D model tracking. We tried image tracking. For a second we thought about a hardware solution. for occlusion and tracking. And that might be a good solution for a longer-term roadmap, but that's almost a research project in itself. Yeah, so we ended up using both the image target as well as world tracking. We do a scan of the entire room, which helps it when it can't actually see the house, the sides of the house. So it's sort of a hybrid solution that we ended up going with. so the projection map is putting an image onto this like white house that's like an actual model that's in the middle of this table and so what you're saying is that you've got a model of that in the like a unity and then you're sort of detecting the geometry from maybe the shapes
but then you're also looking at the actual image that you're expecting to be there and then comparing it to be able to do the location and then once you have that then you're able to put all the other 3D models and objects with all the trees and everything else around that. More or less, yeah. The 3D object is just for the occlusion. We're only using the images of the sides as the targets. I see. So I guess you had mentioned that you actually had to call up some of the engineers at Apple just to kind of figure out. Oh, sorry. So maybe you could each talk a little bit about your background in terms of like what kind of experiences you're bringing into this experience coming in from the development for augmented reality. Sure. Yeah, I actually come from a performance background. I do a lot of projection work for live events and theater pieces. And a couple of years ago, I made a piece with a group called the Builders Association called Elements of Oz, which was, as far as we knew,
one of the first augmented reality performances of the time. We had a few scenes that would have content. Everyone would have downloaded an app beforehand, and there would be a few scenes with content that your phone would buzz. You'd be able to lift it up and actually see augmented content all around you. So we did that. There were some other effects we did on the phones. But that was the first big AR project I did before this. Yeah, so I started my own company this past year called Light Diver. Prior to that, I was at Within for about three years. So I had my hands on several different kinds of VR projects. I was there in the early days back when Within was Verse. And so I worked on Life of Us a couple years ago, and that's sort of a multi-user VR social experience that was also at Sundance. And, you know, about two years ago, we worked closely with Apple to produce a demo that was shown on stage at WWDC, the Apple Developer Conference, that was sort of a Goldilocks story unfolding on your tabletop.
So I led development of that and got the opportunity to work closely with Apple engineers at that time to produce that demo for the stage purposes. I'm wondering if you could make a few comments about the audio since you're using these bone conductance, I guess, headphones. What are they even called? Yeah, they're Trex Air by Aftershocks, these headphones. They call them headphones? I think they call them headphones, yeah. Because it's through your bone of your cheeks. I didn't know if it was a different... So maybe we could talk a bit about that process of doing something that like the bone conductance with the quadraphonic audio and some of the technical challenges that you were faced there? Sure. Yeah. The bone conducting headphones have actually been really, really easy to work with for the most part. The biggest issue we had to solve with them was EQing for them specifically. So the sound designer had to mix all the dialogue, all the effects, knowing what the output would be, which was really valuable. But the biggest problem for
us was latency. We discovered that since their Bluetooth, at least that generation of Bluetooth, had a half-second latency, so we had to shift everything back on the timelines to make it work with that five-second latency and build out a second version of the app to work with wired, over-the-ear traditional headphones, just in case we ever need to switch to that if it got too loud in the room, because being open-ear, they obviously don't block sound. So, you know, in a public exhibition space like this, we had to have the option to close people off if we needed to. And Quadraphonic Sound, we had some issues with hardware getting it working, because all of the hardware that was getting specced for it was for digital audio workstation hardware, which Unity didn't really like talking to, and Windows wasn't recognizing it as surround. So we ended up just using the onboard output from the computer to get that going. Is there any other input that you had? Did you work on the sound at all?
I wrote some of the code that shifts the sound half a second back or forward and that was a tricky juggling process because you know when you go do these kinds of public you know immersive installations you don know what you going to get yourself into. If there's going to be a loud party that they want to have the thing running at, and that's not going to work so well with bone conducting headphones. So we built in as many sort of fail-safes, both in audio and for the state machine logic of the experience itself. What should happen should something that's unexpected but we've accounted for happen? Well, did Peter talk at all about, like, the conceptual reasons for the bone-conducting headphones? A little bit, just for the social dynamic of people talking to each other. Yeah, we covered that. So I'm curious to hear a little bit more about the correction that you had to make in terms So if there's an occlusion that comes off, if the tracking gets off, you need to somehow detect that and sort of either lock it in or slowly adjust.
So maybe talk a bit about that dynamic of if it's something like SLAM, which is the simultaneous location and mapping where you're constantly doing these approximations, or what types of things you had to kind of maybe do with some sort of hybrid or create your own sort of solution that is able to do this self-correction if it finds and discovers that it's off. Yeah, so it is doing constant world tracking to know where the phone is so that you don't always need the image target there. But since the image target is going to be the most accurate, and as Brian said, it's sort of the ground truth, we are always moving our occluding model to that image basically as quickly as possible. So it's always snapping and correcting itself, despite whatever the world map says where the phone is. So we're very quickly doing that, which is why we're able to get that edge as tight as it is. And we are slowly moving the rest of the content to that point because we don't want it to be snapping as quickly because it would be disruptive for the viewer.
So it's slowly interpolating to that position over time. Yeah, it's sort of like the occlusion, the invisible occlusion mesh can snap as quickly as possible, as quickly as it can, because you don't want to see a tree behind the house if it's off by a few centimeters. And it can correct itself and sort of block that tree that shouldn't be visible from that angle. But you also don't want to, it's sort of a user experience distraction. if the ground plane is very quickly moving and snapping. So we added a smooth kind of transition to all the other content so that if you're looking down at the ground, it's not going to be as distracting if something should be slowly moving towards its target. So it was a combination of both fast and slow sort of 3D geometry movements.
and one of the other things we did was if it gets way far off, we reset tracking completely and move the anchor to the corner of the image target because, again, we can assume that that ground truth plane of that image target is what we can rely on. Yeah, and one of the nice things about this experience is we know what the parameters are, so we know that if someone is more than, you know, 10 feet away at any time, if that's where it's detecting where the phone is, we know that's wrong. So we can use that as sort of a fail-safe to reset the tracking. We also did have problems with image targets. So there's some logic in there. Brian built out some stuff that helps us reject bad data. We are using the front and the back of the house as our most trustworthy targets because the sides have some issues. They have more issues, especially with rotation because they're not as large. So there's some logic in there that sort of knows when to trust what ARKit's telling us.
So if you didn't have projection map at all and you just had White House, would it be a lot harder to do the same thing because you're not having the consistency of the image target? Yeah, it'd be practically impossible. Yeah, that's a tough one. You would sort of need some kind of depth sensor. you would need to rely on some depth-sensing 3D geometry, and that doesn't care about image targets or color or anything like that. So that would be more of a hardware solution. Yeah, I think there would be some techniques that might work. You could use a sort of built world map. It would help, but as soon as you get down there and it's just looking at white, there's not a lot of information that it would be able to build a positional database from. So that would be problematic. There are some solutions that use, I think, V4A as a way of just doing objects without any texture information, just based on the relationship of the planes. But our house is too simple for it to be, I think, for it to be able to really gather any valuable information for something like that.
So it sounds like that it's the hardest to have nothing at all, just completely white. And it sounds like the protection map is probably still really difficult, and that maybe easiest would be if you had just a consistent image on the house and it wasn't changing at all. Yeah, if we could have something that was unlit that didn't receive shadows, that would be great. But that's a creative decision, and that didn't work in this instance. But we work with what we're given creatively to kind of try to find a solution within those parameters. It seems like because of that, because you know what the image is and it's constantly changing your dynamic, you have to, I guess, update what those image targets are based upon where it's at. And so was there, like, additional, like, a timeline? Or how do you do that sequencing of syncing up what is coming from the projection map into Unity? Yeah, well, in a perfect world, you'd be able to constantly feed it exactly what it should be looking for in real time, right?
But that wasn't possible for us. and we did look into ways of having a number of images per house side, so a house with a little bit of tree on it, a lot of tree on it. But we actually didn't need to do that. We just fed it a single image for each side, and it worked out well enough. I noticed that sometimes as people were walking around, they cast their own shadows onto the sides. Does that give problem with the tracking when people accidentally get in the line of sight of the projector? I mean, it can't find the image target at that point, which is why having the world map as a sort of fallback helps. So it just keeps it where it last found it, and then the next time it sees it, it auto-directs. I'm curious what you see that this now enables in terms of you figuring out all these technical challenges. is what other types of, like, projects could you now do now that you've sort of figured out this tech stack?
Like, here you have a model on a tabletop and you're doing a projection map. Like, what else could you do now? Like, protect on the wall or what else could you imagine that would enable? Yeah, I mean, there's a lot of different sort of stories that you could tell in this kind of installation. AR space. And if you look around at Sundance this year, there's a lot of Magic Leap installations, and it's kind of really hard to distribute, at least at this point, a headset-based AR experience. so it sort of depends on like where this is going to be exhibited if it is eventually let's say it were to be an at-home experience then you don't get all of the sort of extra advantages of you know three projectors
and knowing exactly the physical size of the thing that it's living on So for the installation space, this was an experiment in a lot of different types of technology, and I really loved kind of being able to sort of rely on knowing that the house was, you know, 12 inches across or whatever. And that lets us sort of say the user is always going to have, hopefully, as close to the best experience as we can provide. So technically, it's been really great sort of doing this as a pop-up kind of immersive installation rather than an at-home experience. But eventually, in the next couple of years, once everybody is just wearing AR headsets, we're not going to be solving for the same problems. So I think once some new technology comes out, that sort of consumer-grade AR headsets will be solving different problems.
Yeah, I mean, I think, you know, the sky's the limit. I really want to see what would happen if we'd use this at a larger scale, you know, like either building scale or having, you know, a space that you can move around a lot more in. because, I mean, this has really sort of opened up my way of thinking about how a lot of this stuff works in terms of, like, hybrid solutions for tracking and how you can have, you know, digital assets that represent real-world assets, especially for occlusion, things like that, and thinking about what sort of the constants and the variables are in experiences like this. Like, it's really nice for installations and experiences to, we know exactly the hardware we're working with. So for, like, an at-home thing, we don't know what people's phones are. We don't know what their headphones are. But here we know what the space is. We know what they're going to be using. And we know how to optimize for it and how to figure out these little, like, hacks and workarounds to make solutions that might only work for this.
but are unexpected to people who know what the traditional solutions are. Well we here at the New Frontier Central Space and they have a giant immersive stage that has a whole bunch of projection map that happening with very dynamic scenes that are happening right now And a piece that I think is called Anal Mosh, which is kind of an unfortunate name. But anyway, so I'm curious if the technology that you created here for the projection map could be used on something at this large of a scale that you're, like, walking around and you're able to maybe have unique enough of an image for you to geolocate yourself where you're at. If you could imagine being on an immersive stage like this with this type of projection map and be able to create a whole augmented reality experience within that. Yeah, we're actually using the exact same software to map our houses that we're mapping this stage with. And I think we might be using the same brand of computer, too.
So, yeah, so you could use exactly the same tools to do this and that. One of the exciting things is that we're, since we know where the phones are with the image targets and with the world tracking, putting it on a stage like this, we could know where every audience member is in real space and do effects that really respond to their position so that not only you get augmented content on the phones, but you have something appearing at a much larger scale that everybody can see. Yeah, I think that kind of experimentation is really interesting because then you can dive into these kinds of social experiences where everyone in the audience has an iPhone that can use AR or if they don't, you know, in a couple of years, most people will. And then you can sort of see where other people are virtually relative to your physical position. So they become part of the experience. And so a lot of what it becomes is designing around these social interactions and how should a user interact with the piece? Are they walking around and does the piece change
based on their physical position, or is it more, you know, just you're seeing like the sims pop up above their head, and even something simple like that might add a level of immersion that we haven't seen so much of. Yeah, the first time that I did the dial, I was the navigator, and nobody in my group talked to each other at all. I think someone maybe pointed at something at some point, but it was pretty nonverbal. But the second time I did it, the other two participants were starting to talk to each other. And so they were starting to, you know, use the space to be able to communicate. And because they had the bone conductance and not headphones, they were able to actually do that. And so, but I'm curious if there's any other things that you had to do in terms of the technological infrastructure in order to either encourage or facilitate these types of, like, social dynamics and interactions. I mean, I think the scale implies, I don't know, it doesn't, yeah, I guess it implies a certain interaction with what's drawing you in.
And I think that sort of relationship to have a small thing on the table and your phone as a sort of a magic window almost. and the only sort of other things at normal human scale being the other participants in the room, I think, encourage that sort of interaction. And also having the phones instead of a headset, I think, you know, encourages you to sometimes look either over the phone or out around in the room rather than being locked into something and having, like, basically a mask on. yeah i mean the the hard part was not really knowing until we got into qa and started throwing a bunch of people into it how they were going to interact um were were they going to just run around the table um and uh sort of end each season of you know experience early and we sort of had to account for all these edge cases. What happens if someone runs around the table while there's
a scene still happening? So we did account for that and built in some kind of nice ways to transition between scenes. But yeah, it wasn't until QA that we really got a chance to see what people were going to do. And it turned out most people sort of wanted to see the content and walked around the table slowly and watched the scenes play out. And everyone sort of got a slightly different experience, but it worked as intended, which was great to see. Yeah. There is a little bit of, there was a little bit of wondering how people, as Brian said, would react. And we tried to structure the dosage speech in a way that sort of guided people towards what we wanted them to do. And we also built in the sort of prologue as a way of giving people an opportunity to sort of kick the tires before the story started, which was important to us because, you know, like the first thing you do with any sort of interactive thing is sort of wave your arms around and sort of see what happens.
And we wanted to make sure that people had the opportunity to do that in a way that wouldn't disrupt the narrative. Yeah, it was almost like you're telling them the rules, but you're giving them an opportunity to have a very fast iteration of each of the seasons so that they can kind of learn, I guess, the pacing and the rules of the experience so that as they go around the table, they kind of see that they're triggering it and then know what to look for so that they can, you know, I guess you're teaching them how to actually operate and be the navigator and see how the experience works. And when I did it, at one point, I went backwards once, But it was the last thing, and it sort of ended. I was like, oh, shit, did I arbitrarily end the experience early? And then when I did it the second time, the navigator didn't go backwards at all. And so do you find people go forwards and backwards a lot? Or I'm just curious if there's some people that start to really remix time in that way. Here, people have mostly gone very linearly. They start going around the table counterclockwise and then just do that three times.
and then it's over. When we were doing testing back in L.A., we saw a lot more experimentation, a lot more people moving around. Yeah. Cool. And I'm curious to hear from each of you, what you want to experience in augmented reality. I'm excited about social interactions in AR and sort of what comes next. And, you know, I want to see the next consumer-grade AR headset that is just, that can provoke these kinds of social interactions because everyone has one. And it's just like looking at your iPhone, except you're looking at your whatever, looking through this device that's projecting light into your eyeballs. So that's really exciting. And not even just on the AR side of things. I think there's sort of another discussion around social communication on the Internet
with some of the discussions around privacy and Facebook and all that. I think there's room to rethink how people communicate online and what that looks like into the future. If this were an ideal world and technology wasn't a barrier, how would we want to interact with strangers or friends or meeting new people? So that's something I'm thinking a lot about these days. Yeah, social is really exciting. I want to see something of a much larger sort of conceptual scale. I mean, a lot of the stuff we see is demos in a lot of ways. And I want to see something that is world scale or is multi-user, something that uses ARs more than just a gimmick and really puts it first and makes it a key part of the concept.
I'm curious to hear what some of the either biggest open problems that you're still trying to solve or open questions that you're trying to answer. Yeah, I mean, tracking in a dynamic situation like this is probably the biggest. It was kind of our, we kept coming back to it. It was a constant nag for us, and someone kept having to fight. So something that makes it a lot easier, that is sort of a plug-and-play solution, would be really valuable. and how to make occlusion work better, have it dynamic, so you can actually, like, either hold your hand up in front of something and it finds exactly where it is, that sort of thing. And, like, getting world-scale tracking good. That's exciting to me. Yeah. I mean, I don't want to have to solve the kind of AR cloud solutions that other companies are investing lots of time and money into solving
and are doing a really great job at it from demos that I've seen. Like 6D.ai has some impressive demos, and I would love to work with them on something in the future where it sort of integrates this AR cloud solution where it's a larger scale AR tracking system where if someone anywhere in the world has ever sort of pointed their phone at a table, the AR cloud should be able to know that, should be able to download that and relocalize against that. So someone's going to do that. I don't know who's going to do that, but all that's really exciting to watch happen in real time. One of the things that I think about both VR and AR and the difference between the two is that I see that virtual reality, you're able to completely unveil yourself into a virtual world and completely shift your context into a new context.
And that AR, it's very difficult to be in a context and just completely shift your context because you're still in the center of gravity of that context. But I feel like the advantage and the difference of AR versus VR is that AR is going to have to try to determine what that context is, whether it through object detection or being just contextually aware of where you at where your geolocation is But that context seems to be an important part of knowing what types of things that you can overlay on top of that So I just curious if you thought about this problem of context and how you start to address it when you start to try to think about the future of augmented reality and what information you're going to need in order to do some more dynamic computing that's really specific to that person's context. Yeah, totally. I mean, the solution that you've seen here is people doing physical build-outs to define it. You know, like we built a big plexiglass cube. So we determined what that was going to be. I think what would be really exciting is as, you know, AR develops
and there are more at-home experiences, that it can adapt to different contexts. There was a piece of sort of a game that was really exciting, came out a few years ago, that basically claimed to turn your house into a haunted house. And so it was able to scan the walls and place sounds that would be changed depending on, like, what walls it's bouncing off of. And, you know, it's a little sort of scavenger-hunting thing, it seems. Things like that are really exciting. Things that can learn what context they're placed in would be really exciting. Yeah. I mean, object detection and human segmentation, you know, it sort of needs to know a lot about the space. but not just the space, but trying to figure out what are moving objects and what things are static. So if there are people moving through the frame, whatever's doing the tracking needs to sort of discard that. Or if it's a piece that's more integrated with detecting that, then it needs to do the opposite
and detect when there are people in the frame and relocalize against a face target or a body target or, you know, skeletal tracking. So there's a lot of different technical challenges with, you know, segmenting 3D geometry, especially on mobile right now. So it sounds like you're using iPhone. Maybe tell me a little bit about the model that you're using and why you decided to go with ARKit and Apple rather than the Google and ARCore. Yeah, we're using iPhone XS Maxes. and a lot of the reason to go with iOS was comfort and ARKit having the features we needed. The reason we went with the XS Max was because it is the fastest phone on the market that could do that and we have been running into performance issues a lot. World tracking is expensive as well as throwing all the assets we're throwing at it
so we needed the fastest thing we could get. And the XS Max has a larger screen. There's a lot of back and forth about what the form factor we wanted the users to have. And we think that an iPad would be too large. And the XS Max is a nice medium between a 10 and an iPad. Yeah, it's really fast. And it's good size. That's pretty much it, yeah. I'm curious if you ran into battery management issues. How long can you run this on a phone? And do you have a number of fleets of phones in order to keep it charged up? We thought we would have more problems than we did, honestly. We have 10 phones back there, so we've got three sets and a spare, so that two could always be charging. But we actually all need one set. We obviously can't run this for a day, so we have to always have them on the chargers. But it's actually been really good. and at the end of the day, all of our phones are full charge.
It hasn't been bad. Wow, that's really impressive, actually. I expect that the batteries will be drained a lot faster based upon what I've seen in other devices. And finally, what do you each think is the ultimate potential of augmented reality and immersive and spatial computing, and what it might be able to enable? Broad question. I think that there's going to be a lot of opportunities for education. And I think that the ability to place virtual content on real content is going to be really... There's a lot of opportunities for that. I mean, I don't know. There's a lot, right? You can do a lot. I'm interested to see how hybrid forms develop that cross the lines between AR and VR so that you could seamlessly go from no virtual content to augmented virtual content
to being completely immersed in a world. And I don't think there's the hardware out there yet to support that. So once that happens and we get further away from bulky headsets and people get more comfortable with it, I think that the lines are going to start blurring between the digital and the real more and more as the frame gets wider and thinner and lighter. Yeah, I just want to comment that the Quest has full VR and has a pass-through camera that I've seen that you're able to do effects onto the world that are pretty trippy. But I'm not sure how good of a pass-through camera is to see a real depiction of the world. and what the IPD is of the camera. Usually it's a challenge if the camera that you're looking at doesn't have the same IPD as you, and so that would require each of the headsets to have the same distance of the cameras as your eyes, which I think would give you more of a one-to-one representation of the world.
Otherwise, it gets distorted and skewed. But I think that the Quest, I think there's going to be some interesting experimentations there of doing, like I did the Dead and Buried experience at Oculus Connect 5, and they had a pass-through camera where I was able to run around the whole space. And it was pretty trippy to be able to run around and move around in a big, large space like that. So I think there's going to be some experiments like that, but it's not going to be the full where we need to be to have a more seamless AR, VR blending like that. But what about you, Brian? What do you think the ultimate potential of augmented reality might be? The future is social, volumetric, and wireless. So all the big companies are trying to solve geometry. How does it exist in a space? How do you relate virtual content to real-world content? Is it happening? Is it always scanning a space? And some of this is going to be a combination of techniques. You know, maybe given the experience, you want to be able to remap your room in your own way.
And so it goes back to the design of the experience. How do, given the technology that you have, what does the user want out of it once it's like a consumer-grade device? And it is wireless and, you know, cheap to buy one of these headsets. And it could do AR and VR by choice, right? So you can hit a button on the side of your temple and turn off the real world if that's what you desire as a user. So there's really a lot of UX questions around what should this future cloud-driven, volumetric social world where everyone has these crazy future Blade Runner Ready Player One headsets that they're walking around in, and what are the safety concerns and privacy concerns. and there's a lot of questions that have yet to be answered, and I'm just excited for what comes next.
Well, just a quick follow-up on that, because I was at the dev lab at Riot, which is owned by Verizon, and they gave a whole talk on 5G, and so I'm just curious to hear your thoughts on the coming of 5G and that if some people that are in this immersive space are really hoping that we have this kind of metaverse-like behaviors by which we have the power and the speed of 5G, but what's it going to be used for? And some people are like, well, you could start to download really quickly these volumetric content overlaid on top of the world, and that would seem to be a pretty good use case for using the full potential of what 5G has to offer. But I'm just curious what you each think of if that's a viable path, if it's just filled with hype, or if you're excited, what the potentials of 5G is going to be bringing. Yeah, I mean, bandwidth to download volumetric data or high-resolution imagery is going to be great for content creators. I think it's going to be, yeah, I think it's going to be really, really powerful.
And I think especially enabling things like this, you know, world-scale cloud-based database for doing tracking like the 60 is doing. 60, right? 60 AI. 60 AI, yeah. Yeah, I think that's where it will come into effect. And I think that's going to make it much more accessible to more people, you know, as it becomes more pervasive. Okay. Is there anything else that's left unsaid that you'd like to say to the immersive community? I mean, thanks for making great work, and I'm excited to see everyone's pieces here at Sundance. I haven't gotten a chance to yet. So everyone out there who's listening, connect with me, and I'd love to chat about your pieces. Yeah, it's exciting. There's a lot of really good work here. Awesome. Great. Well, thank you so much. Yeah, thanks. Thank you. Thank you. Thanks so much for listening to the Voices of VR podcast. And I just want to say that I'm really trying to live into the transformation economy, meaning I want to give away all this information for free,
make it publicly accessible for not only you but the entire community. And if you find value in that, then please do consider supporting me on Patreon. I'm on this journey of trying to reach my 1,000 true fans, And if I have just $5 a month from that $1,000, I'll be in a much more solid and stable financial position. So I really need your help to be able to get to that point. So if you enjoy what I'm doing here at the Voices of VR podcast, then please do consider supporting me at patreon.com slash Voices of VR. Thanks for listening. Thank you.
番組の概要欄(原文)
I interviewed Jesse Garrison & Brian Chasalow about The Dial at Sundance New Frontier in Park City, Utah on Tuesday, January 29, 2019. This is a listener-supported podcast through the Voices of VR Patreon. Music: Fatality
出典・この記事について
- 番組
- Voices of VR(Kent Bye)
- 元の回
- #1791: [2019] The Techical Achievements of “The Dial”
- 配信日
- 要約作成日
- 公式サイト
- voicesofvr.com
この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について