耳まわりのジェスチャー：イヤホンで操作するVRとウェアラブル

Welcome to Data Skeptic Human-Computer Interaction. Today we're going to get into head-based wearables and gesture design. I speak to Shimon, a Ph.D. candidate at the Sharrington School of Computer Science, University of Waterloo. We do a quick tour of head-based wearables, what kind of options are out there, and what the state of the art is. Then we focus quite a bit on his flagship work, a gesture-sensing earbud that reads swipes and taps beside your face. If you've ever done any VR, you can imagine how useful that could be. And if you haven't, we'll elaborate more on that idea. My name is Sheikh Sean Arifin Shimon, and I go by Shimon. I'm a doctoral candidate and a PhD researcher at David Sheraton School of Computer Science at the University of Waterloo.

And I focus on human-computer interaction in my research area. More specifically, I focus on how users can design new gestures for head-based wearables. So that's like the overarching area that I'm focusing on. data type. So I think head-based wearables, they're not necessarily new, but when you think about the history of this area, where does it begin in your mind? Well, I'd say the most common two references that we'd like to use is, one, the ear-based wearables, because people have been using earbuds for a very, very long time, right? I mean, earbuds is like nothing new. You've been We're using this since, like, you know, Walkmans, like, when they're in, like, the mid-80s. And as we are talking, we're using these headphones, like, the big ones that cover our ears, right? I mean, recently, in the last couple of years, we've been looking into VR glasses and, like, you know,

Meta's augmented reality glasses that superimpose these informational texts onto the real world. Or you can have your own reality in virtual reality. But beyond that, you also have stuff like smart jewelries. People can wear smart jewelries on their ears or on their head. Head-based wearables, the most important would be the glasses and the ones that are on your ear. And ones that are on your ear can be your headphones, your ear bars, or like regular audio devices for people who have trouble gearing. So, medicinal purposes and like regular basis. So, those are the most common things that you see. And if you want to go into deeper on which is the most common one that you see around these days, I'd say that would be your in-early animals, apple animals. So, that's one of the key areas that I think is my research. Would you say, and I guess this is a hypothetical question because it's about the future,

But I'm curious if you have an opinion on where we are in the way in which the form is going to evolve. A while ago, all of our earbuds had wired. They were all connected. Now we've lost the wires, and the wires probably aren't coming back. But also our ears are one shape. Our ears don't change. And it seems like maybe this technology would evolve into a perfect form or something like that, and then we'll just have it. Do you think that's the case, or is there more evolution to be done with the way in which these devices are defined and built? That's a really good question. There are current focuses that are looking into the perfect core factors, but there are focuses on how ear-based wearables can fit into your ear perfectly. So there's the size of the device and how it fits into your ear canal. And I believe that companies like Apple and Bose are looking into how these devices themselves can fit perfectly to the mold of your ears.

So people are right now focusing on two things. One, the form factor itself and how it touches head and like can a more perfect molding be done to individual shapes. So on the topic of if the overarching form factor is going to change, I don't think that the shape is going to change that much on the connectivity. We have moved more on to the wireless, but I think the next evolution in the form factor would be to molding to individual needs. because although we have a common ear shape, different people's ear canals can have different shapes and like the in-ear buds that we use, it can be made into a more perfect mode. It can happen that you can go into the Apple Store, they do a scan of your ear canal and they give you that perfect ear buds

that can fit you in. So I think earbuds are pretty ubiquitous. I have a pair that I wear more during the day than I don't, I think. But I don't want to take for granted that every listener knows the experience, or maybe they have some knockoff ones or something like that. Could you describe what you think is the state of the art in terms of what these devices can do besides playing audio? That's a really good question. If you look at the recent devices, I'd say the mid-priced to the higher-priced one. So let's say that you're looking at Samsung's earbuds that are optimized for the Android, Samsung use or like the Apple earbud. They not only come with audio and I'd say noise suppression capabilities, but they also have additional features such as physiological sensing. And that is where people are looking more into it right now. So it's not just about giving you that perfect audio experience. It can also like a lot of monitoring on how you are like, you know, breathing and what's your blood pressure is looking like because it maintains constant contact with your body.

You know, a lot of the sensing that you can do on Apple Watch, but the Apple Watch is on your hand. And if there are physiological sensings that is more specific to the above neck region, earbuds and smart glasses are the best. positioning to that. So the recent thrust by the manufacturers have been to expand more into that area. That's one. And the second thing is that people recognize that these head-based wearables have limited physical space. So how you can expand the area, right? So another thrust is that, you know, how you can do non-device sensing. For example, with Apple Airpods, you currently have I believe, head nodding, so that you can give additional input without touching the airport itself, right? These are the additional features that people are looking into, like the manufacturing side,

so that the users have more things with the head-based wearables, besides just having the smart glass experience of just like listening to them And what are the types of research questions academics like yourself will ask about this area Well for example there are multiple facets to that research One purpose of the research could be if you had sensors like accelerometers and gyroscope which come off the shelf in like standard Apple Air Force, how you could leverage those sensors to do physiological sensing and like, you know, medical monitoring. That's one. Second is that if you had those sensors, would you detect non-touch gestures? Like for example, head nodding, jaw chewing, or touching on your like skin at different parts and like recognizing those. Another trust would be if you could do that,

What are the possible application cases that you could look into? For example, in my most recent research, I took that research question that let's say that you have a linear earbud, which is research way to say that for a factor like Apple earbud, a earbud. If you have a earbud that's capable of gesture sensing and you use that with a VR headset, right? If that earbud had gesture-sensitive capable of detecting media and on-screen gestures around the earbud, how could you use that in interacting with VR? With the MetaQuest headset, you have bare-hand gestures, right? You can, like, use in front of your device. But could you use that gesture-sensing earbud without changing the VR hardware itself to give it more capabilities? This is the kind of research questions that we looked at. And we basically showed prototype of game where you can control continuous motion using gestures around the earboard in VR.

And like we transition between gestures in front of your VR, controlling the movement using around ear swipes. So those are the kind of research questions that we ask in the research space. on, okay, if you have this capability, how could we expand it? For the capabilities itself, how could we expand it using the existing hardware, and can we add more hardware onto the earbud or the headset to give it more capabilities? So these are the three out-of-their segments of research questions that we ask. Well, I'm wondering if you could be the first-person view for the listener and describe what that experiment was like. So if they sat down to be a participant, they put the goggles on, what do they see and what are they asked to do? They're basically looking inside, I'd say, a VR environment where they're asked to run through a maze. So the maze has different turns, different obstacles that you can jump over, can crouch under.

You can pick up different objects like using gestures, audio controller. So what we have to do is to do a full range of motions. So you are put inside an environment where you can walk forward, walk backward, crouch, jump, crawl around, and like become different objects. So your motions are varied, and you perform that using the standard VR controllers and their hand gestures. So with Metas and some other VR headsets, they have bare hand gestures supported, right? You can make gestures in front of your headset and you can grab two different objects and interact with them. So in the resource space, they basically like looked into using some of these front of the device gestures for movement control as well. Now, why is that important? think about the regular VR user who likes to use VR from the comfort of its home.

You are sitting on your sofa. You're just relaxing. You don't want to, like, you know, move down. But you are playing an exploration game where you have to, like, you know, walk around and look in different spaces. So in your home, you don't have that amount of physical space to always use your normal lens to do the actual, like, walking around your home. When you are sitting down, it's easier for you to use controllers. But if you want to get the VR experience, you want to do the hand grabbing and pulling, so you have to put down the controller and you have to use the hands. So if you want to move around and you also want to use your bare hand, you have to switch between using the controller and not using the controller. If you're sitting down, that's one of the next stages. So researchers looked into using bare-hand gestures for everything, but previous researchers, when they tried to use these front-of-the-device gestures using their existing paradigm, what they found out was you have the front-of-the-device gestures

for movement control, but it also includes your viewport. So you might not be able to see some of the things that you're seeing in front of you. Or it can produce conflicting commands. So you want to use two finger gestures to indicate that you're walking in front, but your device may end up confusing it with a grab gesture. So here comes where we look into things. Taking an imaginary earbud that's capable of recognizing gestures on the side of the face. So when you're on the side of the face or on the ear, that's completely outside the field of the view of your eyes or the rear headset. So if you're making hand gestures on the side of your face, it has absolutely no way of getting confused with the bare hand gestures in front of you. And you can use these gestures on the side of your face from any posture. You're standing, you're sitting, you're lying on your bed, you're relaxing, you have a lower limb disability, it doesn't matter.

You can just raise it, as long as your hand is working, you can raise it like on the side of your face and like make gestures around the earbud. like a command to the rear headset as long as that earbud is gestured. We had the controllers. We had the previous paradigms where people used front of the device gestures for locomotion control or movement control. And we came up with this earbud that's capable of setting gestures on the side of the face. And we asked them to use that for controlling the movement and when they wanted to use anything in front of them to just take their hand in front of them and do the gravity issues. So the motion was full. And we compared these four approaches to see how comfortable they were to be. When you talk about that comparison, what's the measurement? What ruler or calculation can you do to say this one is definitively better in some regard than another? How do you measure it? Based on the metrics of the previous research, we looked into the task completion time.

We divided the play area into different parts where we measured the time between sequential tasks and we measured them So like you know when we measure the time between sequential tasks and we give people the same set of sequential tasks comparing that we could measure how fast people were doing the tasks themselves That's one measure of, you know, metrics, and also what we do in VR, have a standard set of questionnaires on VR experiences that we give people after each approach was performed. So they did one approach, say, for example, then they filled out a set of questionnaires that how comfortable they were doing on these types of agile actions. And then we measure the results from those question answers. There is also another metric that I use in these kind of researches

is the physiological metrics to measure the stress level. But for this particular research, we didn't do that because those approaches came out fairly recently. At the same time, we worked on that business as elements. But mostly, the task completion time and the standard set of questionnaires, like what we call SSQs, and I'll actually have to look at the full form of those questionnaires as well. But, yeah, it's the same question, yes. And just, I guess, a question about experimental design. I don't know if you had the same person do different techniques, so then it seems like, well, maybe if I play the game a few times, I'm just getting a hang of the game, and you could have a bias towards the last performance, so to speak. Could you talk a little bit about how you account for that in your data? This is a great question. And in HCI research, we always try to mitigate that kind of, like, you know, bias. So what we like to use in HCI research is something called a Latin square.

What we do is like a within-subject design where each of the participants perform all the conditions, but we don't give all the participants the conditions in the same order. So if participant A does controller first, then within field of view approach one second, within field of view approach, second, third, and like the around ear gestures, the fourth time, we give the next person a different order based on an approach called Latinx way. So this also affects how many participants that take on for these studies. It usually has to be like a multiplication of four. But if done correctly, it mitigates the changing of the order, mitigates that kind of biases that we talked about. Well, I have to confess, I have not owned any of my own VR devices, so my exposure is somewhat limited. I, of course, tried them going back to Nintendo's Virtual Boy

up through the meta headset and whatnot. But I guess what I'm curious to hear from you, someone who's closer to this space, is where you feel we are on the development platform. Televisions, on the other hand, I don't need a better TV anymore. It's as mature as I think it needs to be. Where are we at in the VR evolution cycle? That's a good question because in the VR evolution cycle, I'd say that we still have a lot of improvement, the hardware cost being more accessible. One thing that your meta is doing, it's bringing down the VR hardware cost still within the consumer range. But if you look at other VR headsets, they are still more costly on like the 800 to 2,000 spaces. Like those give you additional hardware, cameras being mounted on different side of like the walls and those and things. But from a consumer point of view, you still have a lot of room for improvement so that your cost is still affordable

so that general users can use this as not being a cost-burden thing. So I believe with Meta's Quest 2 or Quest 3, they're bringing this closer to the consumer space, but on the general VR space, still a long way to go. Same thing with the smart glasses, good development being done. Google itself is doing a lot of work on both VR and air glasses, but I do believe that from a hardware standpoint, feature-wise, it's matured. Like with gesture sensing, using the mounted cameras to detect your hand tracking. It's quite mature, but the hardware cost itself is where things have to be improved so that the end users are feeling more affordable. So I think there's some really interesting insights in your work around how you can combine that VR space with the out-of-vision ear interactions that I'm sure people are familiar with, even if it's just tap-to-go-forward kinds of stuff.

I'm curious to hear how you see those insights, I guess, getting consumed into the world. Will it be that developers hear that and say, aha, that's a good insight. I'm going to follow it. Or does there need to be a library that encodes your gesture recognition that people adopt or something along these lines? I think what's happening right now is that what we're envisioning this research to do is that you would have add-on device like a YonBud. That's being paired with your headset. So you don't do any hardware add-ons to the VR headset itself so that you can use this technology on backdated VR devices. You know, VR devices that are already in existence. So the only thing that you need to worry about is doing a wireless connection with your earbuds. and the EarBird hardware itself is going to have the software to detect gestures around your ear or around the face.

So once you have a wireless connection using Bluetooth that detects those gestures and sends this to the VR headset, the VR headset is just going to have software library to decode that and translate that into whatever gestures that you need to do. That's the kind of implementation that we went for. So, there are two aspects of this. Back in 2023 or 2024, a group of researchers from Switzerland, either EPFL or ETSU, they came up with a year-born hardware that mounted millimeter wave radars on the year-born itself. And what it was capable of doing is detecting these media gestures around the year-born that we were talking about. And not just media, like the all-scale gestures around the earbud itself. So it was capable of detecting and translating those gestures into different classifications. So we just needed like a converter. When we saw like that hardware in existence, we just thought of like, you know, taking that and using it in the VR-like concept.

It's not just about the game developers. It can have like multiple applications. You want, let's say, a person who has a lower limb disability. get real interaction sense in VR. In not the controller sense, but he wants to do grabbing, pulling in VR environment. But because he's confined to a chair and he can use his lower limbs there is no way for those people to experience gains that involve movement So for them this approach would be a real good one You can use hand gestures to indicate them to pick things up, but how do you control the movements without the control itself? So that's where this research can come in. And the most important part, you can use to control the movements from any postures, standing, sitting, using it from the comfort of your couch on the comfort of your bed. That is what I believe is the most important part of this research.

Giving people the experience of bare-hand virtual reality and not be confined to any static postures. Let it be posture agnostic. How you can use it is on your imagination. That hardware from the Switzerland people and with the software package that we have, we already have a solution that gives you their hand gestures giving you all the speech. And we actually have a prototype online. Well, I'm a big fan of keyboard shortcuts whenever I can get them in an application. And I can remap them, you know, if I don't want it to be Command-A, I can make it Command-B. But I'm still working within the slots that the designer gave me. Do you envision the same format that I have to kind of learn an interface? Or is there a world in which the interface learns me. My research focuses on exploring the gesture set from a user perspective. So looking at what the gesture design should look at from the users.

I collect data on how the gestures can be designed for different tasks. So once that's done and we have like a set of gestures, we use that user-informed mapping for the gesture set itself. So, you can think of gesture set and you can map it to different tasks yourself. But if you're like talking about designing completely open-ended, customized gestures for yourself, I believe there is a long way to go. But, especially because we have a prototype of millimeter wave radars being mounted on earbuds themselves, we have the hardware technology and the machine learning capabilities to back up the customized gestures that you're talking about. Now, developing the framework so that you can completely design new gestures on your own and have that recorded and mapped to different tasks,

that's an area that we still need to work on. Well, the human hand is incredibly robust. You could make probably an infinite number of possible gestures, but there's also dexterity, right? If it's some very intricate thing, not every user can do it. Is there any good philosophy here, unlike a vocabulary of gestures? Interesting. Have you even noticed that different hand gestures mean different things in different cultures? I think the middle finger means something different in other places. Yeah, it does. Middle fingers and thumbs up. Yeah, thumbs up for sure. I remember this scene from, I believe the movie was called Inglourious Bastards, where people indicated three with different fingers. So the regular English phrase, three fingers in the middle, and the German phrase, your thumb and two fingers. Ha-ha, and that's how he was revealed. Exactly. So you remember that scene, like in the cafe, right? So what it means is that for indicating the same task,

You may have cultural context ingrained into these kind of hand gestures. So dexterity, the capability of your wrist does come into play, but the cultural context when you are designing gestures also comes into play. And this is why when we look into a suitable set of hand gestures for proposing in this space, we run user studies. And that's what I've been doing in my PGA research. You take users from different backgrounds and you ask them to come up with gestures and you ask them questions about why they are not proposing these gestures so that you understand what are the common gestures people are proposing from different backgrounds for a particular task and you understand the user agreement. If you have a functioning product but you don't like the gesture mappings and you can't use them, you won't be using that product that much.

So that's why my research focuses on the user-preferred aspect of designing these gestures rather than the hardware-focused things that other researchers, my counterparts, are doing in different parts of the world. Well, what's next for you? So my background is coming from a consumer electronics background. So I started my job from Samsung electronics, focused on mobile phone development, and I moved on to like this wearable interaction stuff. So next for me is focusing more on the hardware and the machine learning aspects of head-based models so that we can take these ideas and concepts into a mass consumer scene. Apple, Huawei, I believe Samsung are doing a great job of that. So probably spending a couple of more years in research and then focusing on these next generation of variables

can bring this technology to the mass market. Yeah, it seems like an exciting time to be in the field. It is, and it's also really, I'd say, challenging because in the last couple of years, we have seen an exclusive growth in machine learning and AI research. So because you have growth of LLMs and those kind of things, the research funding and the current focus of the market is, I'd say, more on to that segment of the N-like areas, focusing on LLMs and data centers and hardware. So focusing on these things are a bit, I'd say, not on the market perspective right now. So that's a challenge that one has to balance as well. So the market wants and be such a good interest. And is there anywhere listeners can follow you online? That's a good question. I do have my website. It's called www.shaonarifing.com. If you'd like to look me up online.

Honestly, I have been taking a break from Facebook, but you can always follow me on LinkedIn. You can do a search for my name, Shesha Rafishon, and you can find me on LinkedIn. And most of the work that I do, from my research or like docs and like others, can be found on my LinkedIn thread. Sounds good. We'll have links in the show notes for listeners to follow up. Shimon, thank you so much for taking the time to come on and share your work. Thank you so much for your time as well. I look forward to catching up again if something exciting comes up. and I look forward to listening to this podcast as well.