Regression to the Mean Girls：野球データ解析とBART（Sameer Deshpandeと語る）

This is Regression to the Mean Girls, where we chat with brilliant data people about their silliest side quests. We are mean as an average, but actually very nice. Come sit with us. I'm Lucy D'Agostino-McGowan. And I'm Sarah Lotzbeach. And today we're really excited to be chatting with Samir Deshpande, who is an assistant professor of statistics at the University of Wisconsin at Madison. Welcome, Samir. Hey, how's it going? Good. Happy to have you. Likewise, I'm great. I'm excited to be here. Yeah. So we like to kick off with a little icebreaker that we call Media Minute. So it can be any kind of media that you're really enjoying right now. My husband and I just finished Band of Brothers for the first time. he wanted me to watch it forever um and so many many years later I got to watch the the whole kind of show it was deeply heavy but very very cool and also we talked about this a little bit

before we started recording but it's kind of funny to me how many people were in that and appear in other things like we were saying that the dad Rufus from the original Gossip Girl is like a pretty key player in band of brothers he's i think lieutenant spire spear maybe um but yeah that's what we'd love to read this yeah i did it i know i know i know later on i guess that's yeah because he's kind of like a that i like logan i like that like the kind of i like logan i don't it's not that i will say though watching band of brothers my brain could not like it was a this classification problem because I was looking at Lieutenant Spear. My brain was like, we know this man. But my brain was like 50-50, this is either Rufus from Gossip Girl or this is Billy Burke who played the dad from Twilight, who I met at a Twilight convention when I was a child. Ha-ha. We're going to circle back to – there was enough in that sentence

that we're going to have to circle back and really unpack. Twilight convention as a child, I know there's like fun. moms and cool moms but like oh boy no my mom didn't go my mom was way disinterested way too cool my mom's sweet friend christine took me my mom was like i'll catch you later christine can take you to the twilight convention oh my gosh i don't know if you are closer to sarah closer to me like generation wise but twilight was just i'm like a little old for twilight so i didn't like but my little sisters were into it so and i think also having little i think there's two things because I feel like people might could have been into Twilight, but the fact that I had little sisters that were into it made me, I missed it. But I could have gotten into deep. I mean, I was like writing Harry Potter fanfics and stuff. So it's not like I was above being at a fan convention for something like this. To this day, the only fan convention I've ever been to. And to brag just about one more element, because this was maybe 2007, 2008, I designed T-shirts in Microsoft Paint,

printed them onto iron-on transfers and had made matching t-shirts for my mom's sweet friend christine her niece found me which um not to flex but kellen lutz who played emmet said quote nice shirts so guys i peaked then i think that what could be better at a fan convention i'm shocked this is the only one you went to the adrenaline rush of like attending a fan convention around making a shirt ironing it on people and then to have someone like someone with the level of i don't know this person because i don't know the twilight people but that yeah sounds like this was someone who was involved in twilight's complimenting said t-shirt i'm just envisioning your soul like leaving your body um right there but absolutely i would just never stop going to such fan conventions I think that would be just the beginning of the end. What have you been doing for the course of your entire life? Yeah. That was the only one. I don't know why I didn't really stick with it.

Maybe you peaked and you were like, I don't want to. I mean, that is hard to beat. I'd also, I really have to humble myself and admit that I searched my text for the word Twilight. Because I know I've texted my mother in the past to be like, can you send that picture? Y'all, the number of text results. The amount I seemingly discuss Twilight is alarming. And I promise I don't like it that much anymore. But, yeah, that's where we're at. Amazing. Did you read Twilight, Samir? No, never. No. Yeah, me neither. I was a little too old. Yeah. Yeah, I think we're probably, like, around the same. It's definitely in college, I want to say, or, like, about to start college, something like that. 7-08, I was in college in 08, so that would have been, yeah. Yep. I know, we just were discussing this. We were on, I went to Chapel Hill. We were on Chapel Hill's campus this weekend. We took our kids to a football game. We were, like, tailgating. How exciting. It was so fun.

But we had this, like, my husband and I both were like, oh, man, the freshmen were born the year we started college, which is, like, that's not a statistic I feel great about. as I told Sarah because we were like they had such cool swag this weekend they were giving away these bomber jackets for the first 4,000 students and Patrick was like this is so cool we want one of those bomber jackets and Sarah's husband is currently an MBA student there and he was like those are for the students I'm like yeah I know we could not have possibly. I mean to be fair my husband is like talking down to you while also being not that different in age. Well yes but he's an actual student. He is that Steve Buscemi what up fellow children Like, my husband is also not a traditional student's age. He just apparently had the super exclusive wristband that you would have needed. But the amount of, like, absolute disbelief of when I said that, he was like, she thought what? She thought she might get one of these as a non-student.

I know. This is actually a nice segue because I've been going to a bunch of Badger games this year. and one of the games I was sitting, I brought my buddy from grad school. He came up from Chicago and we're sitting in the section next to the student section and the overflow section. And as I'm walking back from or trying to like move in this massive humanity, one of the students in my class just sort of taps me on the shoulder and gives me a, what are you doing here look? And so now I am going next week and I've already purchased my Badgeralls. Badgeralls! Yeah, yeah, yeah. The stripes and the overalls. Yeah, the bits. I know. UNC does it too. UNC does it. They were selling them at student stores. Amazing. This is amazing. I have to say, I love an animal mascot. I love that you're Badger. Bucky Badger, he wears a little sweater. I'm on an unfortunate streak. This is my second school with an unsettling humanoid mascot.

And I've said this before, but I'll put it on the record. I don't like a human mascot. It's a man in a man suit. Between the demon deacon and Mr. Commodore, why are we doing this? Why is it a man inside, or a woman, I guess, but, like, why is it a person inside of the skin of a different person? Why can't it just be the person wearing, like, a suit and tails? You know what I mean? Yeah, well, so the problem with the badger is it's just like a giant rat, and it's got, like, in real life, They got these long claws and like their nails are like that long and they are frightening and they don't make nice plushies. So I give these to my like friends who have like small children when I travel to conferences. I'll like bring them something from Wisconsin. And it is just the I try to find the most like Timu looking. Fear inducing like plushies I can find. And luckily, like for a badger, it makes it very easy. We were just saying, so at least a badger is intimidating.

We were saying this weekend UNC was playing Syracuse, and Syracuse, my kids were like, it's an orange. This mascot. It's an orange. And they were like, an orange? This is like a sweet fruit, not even a lemon. Like, you know, what are we doing here? Also, New York, can they even grow oranges? It feels like they're appropriating citrus culture. Like, why do you get oranges? That's right. As a Floridian, you had something to say. You should get oranges. I do. I have a lot to say about it. They don't even go here. They don't even go here. They don't even go here. This is great. I'm so glad we were able to work in mascot talk. He is apparently a lot less scary than he used to be. Sorry. Really? Looking at the history of the Demon Deacon. Oh, I mean, some of these are horrifying. Samira, so good you did sports. We've had one other sports, but we haven't had very many sports media men as we're like. So this is like, and it's appropriate. I feel like this is like, this is aligned. This is excellent. Well, I was just going to have been rewatching Friday Night Lights, so.

Oh! I love Friday Night Lights. So, okay, now I have to look up because I always like to make a Winston-Salem connection. Clear eyes, full hearts, can't lose. Can't lose. There's someone. Okay, hold on. There's a Winston-Salem connection? It's, like, loose, but there is. When I was in high school, one of my friends is, like, kind of a strong word. one of my acquaintances, he went to School of the Arts and he played the, like, rival quarterback in, like, I think it was the first season of Friday Night Lights. We got a Winston-Salem connection. There we go. You really, are you sponsored by the tourism board of Winston-Salem? You will do anything to connect. I just get Winston-Salem in there. I know. But anyways, I'm proud of Winston-Salem. We've got our, we're on the map. Yeah. Yeah. Okay, my media minute. So before we joined, Samir, you let us know that you don't dislike or you like the Muppets.

Yes. I love the Muppets. Which, yes, and I love the Muppets, too. And so here's, I'm going to do two. One is actually going to be, because I have to, like, link into why this Muppet thing exists. But my sister, so my fifth sister, Teresa, she is a musician. Her name is Tisa D'Agostino, if anybody wants to. So that's going to be one piece of media. I'll shout out my sister. She's a composer. She writes piano music, and it's beautiful, and she's very, very talented. But she just got married to someone. Yes, and we're getting to the connection. Her new father-in-law, Bill Sherman, I just found out, he maintained the very first Muppet webpage on the Internet, which I just, like, love this. And that era, the era, like early internet, you know, 90s internet is just like, that's a form of media that I just love. I love finding these like old websites. And I actually, yes, it's all like the HTML, you know, you can go on Internet Archive and find it still, but it's not like still live because it was hosted on.

He worked for like one of the government ones that had a lot of like early internet stuff. I'll put it, I'll drop the link in the show notes. But it's, like, it's one of these very old, like, you know, the text with a couple little lovely pictures. He, like, early on was pulling pictures from, like, people from the Muppets, like, that noticed that he was doing this. Sometimes would just send him, like, they'd send him pictures. So then he would, like, scan them and put them on his website. And then, like, I guess. Amazing. Yes. Someone like sent him like a cease and desist. And we're like, where'd you get those pictures? And he's like, from the people that work at your place. Anyways, but he had like, he had a really nice little blog where he would keep track of all the different episodes of the Muppet Show and Fraggle Rock and Jim Ensenauer and all of those. So that's my, that is my media minute. Bill Sherman's Muppet webpage. The very first one started in the early 90s.

Amazing. Amazing. I know. It's so good. It's so good. Okay. Well, this is a lovely media minute. Let's transition us, Samir, to our context. Can you tell us about what your analysis is? Yeah. So this is going to be totally chaotic and off the wall. But I was thinking a lot when Sarah told me about the show. And I said my first instinct was I love the name absolutely. My second instinct was, my gosh, what am I going to talk about? And then I listened to a couple of episodes and had a phenomenal sense of imposter syndrome and also felt, you know, I don't really have any side quests because all of my side quests just sort of consume my life. and then I realized that is the entire story of the movie and so I went back and traced uh some work that I'm doing now and papers I've written and things I've done with students have traced all the way back to graduate school and doing a thing on the side because I couldn't figure out

how to do anything else and it is just it is it just maps in there perfectly you know just as a bit of context. I've seen Mean Girls about 200 times in my life. There were periods of time where you could drop me into the middle of it and I'd just keep going. You could just speak. Yeah, yeah, yeah. But it's great because it's a super tight movie. There's almost no wasted scenes. Unfortunately, I watched it again last weekend and there are parts that do not age well. uh there is yeah some some of it is real rough uh i'm gonna try not to say any of those parts but uh and it was also interesting for me because that movie came out when i was uh when i was in like the seventh or eighth grade and was super into math contests and i distinctly remember we had a field trip um they used to take us and we're down in texas we did a texas history field trip take us to the Alamo, LBJ's Ranch, a couple of other places.

And getting off the bus, there were a couple of students who had gone and seen the movie the weekend before the trip. And there's a bunch of us who were going to go the weekend after the trip. And I remember somebody was on those. You have to see this movie because you're in it. It's like, what are you talking about? Like, there is a kid who likes math and hip hop. And I feel this is you. I was like, oh, my gosh. Kevin G really is. I really forgot the insane. Yep. please. Yep. Yep. So like, are we trying to get jackets? Yeah, I lived that life for a while. But then to say this, like, I guess for context a lot of my work is obviously in statistics and I do some methods I do a lot of BART stuff so BATCH and additive regression trees But almost everything I worked on started because of a project in sports And I got interested in sports data really as I was a first-year PhD student. I thought I wanted to do a lot of theory, and I was not making a ton of progress.

I didn't know how to start. All I was getting was, oh, read a bunch of papers, figure out all the tricks, and then do it on your own. And that's just not how my brain works. So I wasn't really doing too great. And I had a professor who was who did a lot of sports analytics research. And that was at the time when a lot of high resolution data was just starting to become available. you used to not be able to get things like play-by-play or pitch-by-pitch data in baseball. And that started coming out right when I was in grad school, and some early web scraping tools started to become available in R and Python that people really started to figure out how to do this. And so he said, yeah, if you just want to work on a project, go ahead and I'll help you and show me around. We wrote a paper, and then on the basis of that, that sort of spiraled into another one, which, as all good stories do, it starts in a lunchroom, and we're having lunch in the department. A professor says, have you seen this thing about baseball and catchers

and how they catch the ball is worth like tens of millions of dollars? There's no way this is true. Go build a model and see if it's true. And so I did. And that led to like a multi-year project, got a paper out of it, and things about that project. And it is still very much in that kind of side quest zone. But then I just kind of got obsessed with that problem and kept thinking about it. And at some point I said I wonder if I can try Bart on this problem. This is about you can look in baseball and when you don't swing, the Empire calls it a ball or a strike. And the rules say it's deterministic. Where the ball is determines exactly whether it's a ball or strike. it takes about four pitches of a game to notice that's not what they're doing. It's extremely noisy. There's a lot of variation within umpires. There's a lot of variation across umpires. And there was this theory that players were influencing these called strike probabilities.

And so we built a model. It was this big hierarchical logistic regression model. I had to learn Stan to do it. This was when Stan was just coming out. So there wasn't a lot of tooling and a lot of, there was a lot of reading Google groups and figuring out, oh, man, how do I get this thing to work? It's like 300,000 data points, 150,000 parameters. How is this going to work? And so we did it. And then I said, you know, I'll go and do my real work and do some efficacy things. And when I graduated from my PhD, I said, you know, I have the summer. I'm not really doing anything. I was traveling and whatnot. There was a thing in that paper that we didn't quite do right, And I wonder if I can use BART because BART is this machine learning tool. You can turn your brain off and put all your covariates in and get your outputs and get a good regression function. So BART should destroy this. This would be amazing. And this is the part of my career that's really like that Halloween party in the movie, right when, like, Regina kisses Aaron and Katie has that, like, breakdown.

Yes. Yeah, no, BART crapped the bed. It really did not work well at all. And I said. So now you're feeling a crisis of confidence. A crisis of confidence. Everything I've been told, this doesn't work well. And then I said, well, now I've got to fix it. And that kind of brings us to now, this kind of multi-year arc of kind of something that started as a, let's just go get data and mess around with it, has led to, I don't know, four or five papers, new R package, several years of work, and all for something very, very minor. because I just wanted to predict ball strike decisions in baseball. And it's taken a real – it's had a second life this year, too, because I don't know if you all are familiar with it, but the robot umpire is here now. There's automated ball strike system. They've changed the rules. You can challenge calls. They have technology to really pinpoint whether the ball crossed in the zone or not. And this idea that players have a, like, monetizable effect on call strike probabilities,

I kind of got in at the beginning and it's now a point where I don't think that's a thing anymore. The rules have changed in a way, so I've kind of seen it over the whole arc and obviously happy to go into all the details, but that's kind of where it was. So yeah, Framing got hit by the bus like Regina is maybe how I'll end this here. This is amazing. Well, first, I think that it's always a magic trick when you can turn your side quest into the main quest, just so it seems like you've been able to turn PsyQuest into MayQuest. I think Roger Pang a couple weeks ago had this like great quote about this the importance of like that in-between time like that lunch time that you talked about where you've like kind of had this conversation at first spawned it because it's like innovative things come out of those types of conversations otherwise we're just like doing whatever routine work and it um so it's like I love hearing about people who have been able to turn their PsyQuest into like pull it in. And also to say like culturally there's there's something going on, especially with sports too, because for so long, it really was, if you were an academic

statistician and you liked sports stuff, you had to treat it like a side quest. Nobody hired for it. Nobody cared. You could not get promoted on the back of this. None of the big journals would take sports papers, even though the amount and challenges, the amount of data and the volume of data and the types of problems that come up are incredibly rich. So to give you a sense of where we're at now, maybe 25 years ago, we were still in the box score and reading statistics out of the newspaper. I mean, Efron and Morris in that paper about trinket estimation, they just read the scores out of the New York Times that day. we now have the ability to look at 29 points on the body 60 times a second and like the full 3d pose tracking and i don't know how to i don't know what models to use for that i don't know what methods there are for that we have to make them and so there's a lot of richness in sports

data that only recently i think the profession is starting to uh to do so i would tell anybody who's interested in sports data, like you can make a career out of it. And it really is just a matter of not gatekeeping the profession anymore and kind of go work on whatever seems interesting and not just because you think you can get a top journal paper out of it. I also think it's interesting that historically people were in such opposition because the problems, like the methodological considerations are not siloed in that way where it's like if you choose to approach a problem through the framing of sports because that's what's of interest to you or that's kind of when it first became clear to you then that's super valuable but that doesn't mean that for the rest of time those things can only be used for sports um i feel like i love being kind of thrifty and finding ways to use existing methods in a different application or in a different way um and i also want to echo that

Yeah, making the side quest the main quest, I think, is honestly the dream. I think your side quests were perhaps a little bit more on topic and productive than perhaps Lucy's and mine were, which were just whatever TV show we were dissociating into at that particular time. I don't think I'd push back on that. I think there is no sort of degrees of seriousness here. do the analysis on whatever sort of sparks you join there's no reason to somehow elevate one over the other because whatever you're passionate about that's what you're going to do good work in I also think I love that and I think that to your point I think sports from like an outreach or teaching statistical literacy to the everybody right like Sports is probably the easiest gateway to convince the masses to get into statistics and data things.

And so to just totally ignore it leaves so many people, I feel like, consuming statistics, but maybe not knowing what they need or not knowing when they should be skeptical of certain things. I agree. But, and let me put it because there's a big but on this. if you like sports, it's the greatest way to get into statistics. If you don't like sports, it is incredibly isolating to be around people who are super excited about an application of statistics in this area, and if it's not clicking for you. So I do want to be – This is why you need our weird popcorn. We'll take the people that are into movies, and you take the people that are into – And I think there's a big overlap there. But so, you know, I remember when I was on the chat bar, people were asking, oh, so are you going to do like an intro stat with sports? No. Precisely because for the people that it works well for, it's amazing.

And for the people for whom it doesn't work, it's the worst experience that they could ever be in. And so we do have to do it. We have to be careful. But this is why I think a totally, and I'll use the framing that I, just that we shouldn't be using. I think we should have, like, interest in all the frivolous applications. Yeah. Like, let's do that. Like a standard bag. Lean into the fun stuff. Yeah, show a little bit. I totally agree. And I think part of why, I mean, what I'm trying to figure out how to teach now, and I haven't figured it out, and I think, you know, maybe you both have advice on how to do this better, but I want to teach students to be curious, like, how to ask those curious questions. And I think the easiest way is you just tap into their natural curiosity about whatever it is they're already interested in. So, like, students who are really interested in sports, like, they are going to be curious about, you know, is there this effect of player on whether or not they get called a ball or strike? And, like, students who are interested in Gossip Girl might be interested in, you know, whatever Sarah and I are.

I feel like there are, like, if we could find ways to tap into their curiosity, it makes my job easier because I don't have to teach you how to be curious. You're already curious. And I just have to teach you how to channel that curiosity into the right kind of statistical questions. Speaking of Gossip Girl, there is one thing that I would be kind of curious to try to do, but I haven't spent anywhere near the cycles. And it's hard to do because everybody knows what the ending is. But I'm kind of curious, like, as you go episode by episode, where is everybody's probability on who is Gossip Girl and how does that evolve over the course of this season? Like, how could we somehow pull whatever is happening in each episode? And what? Because you see, like, all of the messages that come out and all of the updating of the stories. Yes. And could there be, like, if we tracked, I don't know. What were people, like, were there a message board? I'm trying to think when it first came out where people would have been putting. Because, like, I don't know that, like, people who watched, when did Gossip Girl first air? I watched it late. I watched it late, too. Like, maybe 2005 was.

Yeah. The internet probably didn't capture it. It would be fun if you could go back and find them. That one, too, has a very historically documented where they did have a plan originally for who was going to be the brother, right? And then the fans figured it out, and so the writers changed it. So that's also, I feel like it's sort of interesting because it would provide, there's this feedback loop of the fans sort of, or the writers felt like the fans figured it out. And so then they changed the story, which I think also kind of plays it. Kind of like that posterior over all the people and, like, when does it concentrate on Dan and when does it like. But speaking about the writers not listening to the fans, I know, Sarah, you have thoughts. And I believe we have the same thoughts about how I met your mother. And, like, I almost I I watched that religiously. And that was a thing that like one of my friends in college and then ended up being a roommate with him at some point. We watched that together throughout college.

And when both of us wanted to put our remotes through the television at that end of the last episode. So just a sorry for that little detour. No, I feel like I still and I haven't watched it, but I watched enough of it. And I just feel like learning how it ends, to me, it feels almost distasteful. Like, you spend the entire framing talking about, like, you're hyping up leading up to this woman, only to then end the show on a note that, like, it was barely ever about her. Yeah. It's kind of how it feels. It ignores all of the growth and all of the kind of richness. If I understand correctly, which I might not, because I did watch it, but I only watched it, like, probably once. But wasn't it that the writers, like at the very beginning, they did have, the plan that they went with was what they intended to execute after season one, but it was almost like they stretched, it should have ended much sooner than it did, but they got a bunch of extra seasons approved in between. So then they had to kind of like stretch this story that then became, like if you watched just season one and then the finale, you'd be like, okay, like that is plausible.

But because they stretched for, like, whatever it was, six or seven seasons, it, like, became no longer plausible because they had to do different things across these seasons. And then it, like, isn't that right? I might be misremembering that that was the original. So at least the Google AI overview agrees with you. Okay. That it was. The AI interview. Well, I mean, I'm not going to actually read. I'm sorry. I'm just going to cite what I'm, like, it's pulling from Movies and Stack Exchange and Quora. So make of that what you'd like. but one way it is synthesized at least that the entire framing explaining the long-winded history of Ted's romantic life was a way for Ted to ask his kids for permission to date Aunt Robin again years after Tracy's death yeah but it was kind of like that I mean the first season to me if I I remember the first season better than the other ones because I probably did re-watch that one a couple times and like that that is not you do get the impression that he likes Robin like the first season that is like yeah I mean really the like whole deal and then it does there's like all this

other stuff they had to kind of put because they couldn't just keep repeating that for seven seasons and then they like kind of I don't know yeah no I agree um I think to your original question Lucy about like how to get the students curious the closest I have found is like doing the final projects in survival where I let them pick their own data is my favorite thing. But there is definitely nuance to like letting them be creative but without setting them a little bit too loose where it then feels intimidating I think it one of the beauties of being statisticians and I teaching statistics classes but at the end of the day we teaching methods and it like a toolbox so it like deciding how we apply them or deciding how we set the students up to apply them you have quite a lot of freedom. So for survival I give a list of like here are recommended data sets you might choose from but I also recommend looking at Kaggle looking at data.world like finding your own data sets and the variety of projects is delightful and you get a lot of a look at what the students

actually like so we met last week Samir with a group from um that took survival with me who did Dancing with the Stars amazing and they looked at how many weeks on the show until like particular stars were eliminated and they looked at did it differ by the sex of the celebrity or did it differ by what kind of industry they were coming in from that was the data set that I didn't even know existed they found that and they were interested in it but a few years before I had a group of undergrads who wanted to do sports stuff and they collected their own data set on MBA career trajectories for people recruited out of like the blue blood schools and so like how long into their MBA careers and like they were administratively censored when they pulled the data um and that is the closest I have found is like kind of setting up the framing of like you have a lot of flexibility here but I'm not letting you totally you're not on an island you don't have to find your own data where you fail but it's like I've set kind of parameters to the project but I very much just

let them pick and you do get a nice little glimpse last year somebody did Chopped like the cooking They made their own chef's hats. I was, like, beside myself. And they looked at whether, like, what kind of position the person was in. Like, were they – Like, an executive chef somewhere or, like – Right. And were they an executive chef or, like, sous chef at, like, a big – like, a major metropolitan or, like, a smaller – like, a Winston-Salem. Like, a smaller city. But, yeah, it is my favorite. That's the closest I think I've gotten. Yeah. I guess the trick there, and I mean, not that I need more classes to teach or the department needs more of my ideas of like what we should be doing, but I think it'd be super fun to just do this as like an independent study. You spend the first six weeks of class asking questions about things they're interested in, because obviously there's the tools and we're teaching them the tools, but we're also teaching them how to translate substantive questions into things that are answerable with data.

and there's many things you can do to some data that you assemble especially in these contexts but it's not always clear that they're answering the question you set out to answer and and trying to teach both ends of the pushing your data through a tool that to me feels like the easiest part it's the is this an question answerable with the data yeah we've got to we've got to come up with a better way of framing that because it's a mouthful. It's like a class in statistical thinking. Yes, I know. I mean, I think that we sometimes it's not exclusively true because there's obviously prediction questions, too. But it feels like I've been thinking a lot about, like, where should we be sticking causal inference, causal thinking, like not causal inferences. And like, I don't need to teach you like the math behind it, but just the like framing of potential outcomes, causal questions, like, what, like, abjective reasoning, what are some other ways that could explain

what we're seeing in this data? You know, that kind of thing I think we could be putting into our intro classes, like, much sooner, like, at the time when we teach you exploratory data analysis and that kind of stuff, like, thinking about the statistical thinking exactly as you said there, like, what is it that we're actually trying to answer? What is answerable with the actual observations we have? And how would that change what we are trying to answer? Or do we just need to find different data? But it's hard to squeeze it all in. But I think that's exciting. If I had to teach kind of the exact same, here's this test and here's when to use it and widgets in a factory, like, I'd go crazy. Well, and that's, like, the kind of stuff that I feel like, especially, I used to teach these, like, biased-out classes to clinicians, and everybody wants the, like, little decision tree of, like, you know, is it continuous, yes or no? okay, now, okay, go down this tree and now do a T-test if it's normal and do a Wilcox, you know, whatever. They want this, like, decision tree.

And that, I understand why people want that because you don't want to be wrong on that decision tree. You know, you don't want to be doing, like, some test for a binary outcome when you actually have continuous data or whatever. So, like, there's a utility to that. But that's, like, totally the thing that a robot or, you know, a non-thinker can do for you. Like, you actually don't, we don't need to be memorizing this tree of, like, things. we need to spend our time on the like statistical thinking part and then the tree of things is something that like you know at this point can be like a plug and chug so i think it's sort of trying to think about what parts we can extract to replace with this thinking since there is a finite amount of time to teach these tools absolutely well i want to hear okay two things when you just described that like sparked my my spidey senses the one you mentioned that um You had originally done a logistic regression, and then you were like, BART is definitely going to do better. And then it didn't. I love this because I am constantly like, we need to, a logistic regression can do,

use a logistic regression with some nice variables, maybe some regularization, put a spline on all your continuous things. It's going to be hard to outperform that. Logistic is very beautiful. And so I like that. The first part that the out-of-the-box part didn't outperform logistic is like music to my ears. But I assume eventually that BART did outperform logistic once you, like, you know, were able to tweak it. So this is also, like, the reality is when you have someone that really knows what they're doing, they can also improve their tools. I want to hear a little more about that, I think. Here's what we did. We were modeling ball strike decisions, and they're fundamentally made by umpires. So what you're not, like, out of the box, we're looking at our outcome is indexed by the person making the call. So you can imagine splitting your data up by umpire and saying, okay, what is the probability that this umpire calls this pitch thrown to this location by this pitcher with this batter and this catcher involved? What's the probability of a called strike? And so the first thing I did is, right, logistic regression, super easy.

I've got the coordinates of the pitch, the horizontal and vertical, as it's crossing the front plane of home plate. First thing I did, straight logistic regression for each umpire. That was bad because, right, think about the log odds of a called strike for one umpire is linear in the X and linear in the Z. So you got this picture of the called strike probability just opening up and out, right, because it's monotonic in all the things. And then we spent about a year really trying to figure out how do you parameterize this space? We didn't necessarily want to introduce a spline because that would get so many more parameters. and we're talking about this and we realized we needed to be hierarchical because you want to borrow strength across umpires, the places where this umpire hasn't seen any pitches. So a hierarchical thing makes sense. And fundamentally what we ended up was a model that had all of the one-way interactions. It had the one-way interaction between umpire and catcher, umpire and batter, umpire and location. We cheated a little bit and feature engineered by fitting a gam to the previous season.

just location and then transformed it so it was a kind of two-step model as a purist i didn't really like that because the uncertainty wasn't being propagated i had to do a step first but fine we were kind of shrinking to what everybody did the the previous season the trouble is when we showed this to people in baseball they said right we kind of all suspected that some catchers have an effect on some umpires but not all. So there's some catchers on some umpires but not all. They also said that we actually believe some catchers are good at framing. This is what is called framing a pitch, you know, converting a ball into a striker or influence that calls strike probability. That happens some pitcher, some but not all catchers can influence some but not all umpires in some but not all locations. So now you needed this like three-way interaction. I mean, yeah, a tree is going to, this is like, because I think I talked, this is so perfect.

My students, I'm going to have to like send them here because I'm teaching statistical learning right now. And I gave them this long spiel last week. We're about to get into regression trees. And I was like, I think that the world is additive and we need nonlinear terms, but I'm not convinced the world is multiplicative. And trees basically are, because we need these multiplicative effects. And also because we have regions, like a tree is going to, it will not actually be the same thing as a three-way interaction because there's some regions that we don't actually, like that don't need that extra, you know, which is what you just described. And you just very eloquently talked about a domain where there are clearly multiplicative effects. And so there's like, you know, I'm mostly thinking about like in the healthcare space. And certainly there are multiplicative effects in healthcare space too. But I think this is, yeah. It gets even richer because, you know, you could do the, you know, didn't plate spline or whatever, expand it out, have it varied by umpire, add all of these terms then, put a sufficiently regularizing prior. At the end of the day, you're getting all these parameters, none of which has a substantive meaning.

Because you only care about the predicted probability and the counterfactual, what would have happened had there been another catcher, right? So there's so much of this where we're fitting these models, and we're just going to – I was doing an innovation where I said I get these posteriors, I turn them into posterior predictives, and I throw away all the parameters. So you're doing all of this sampling work to get these parameters that you're never going to look at again. You have no way of interpreting it. So I said, right, throw it into BART. It'll figure out the interactions, where you have player, umpire, location interactions. You can put more. You can adjust for more things that count how long of the game is it. Is the game virtually over? And the umpire is just like, let's get out of here, hurry it up, or high leverage situations. So you can put lots of stuff in the tree. And then I thought the probabilities were actually a little, the predictions were actually a little bit worse. And I said, well, this is a problem. And it turned out the issue was how BART treats categorical predictors. And Sarah, you've heard me waffle on about this years and years ago,

but this is all part of that larger ARP is what do we do in a linear model? We just one-hot them. We introduce those binary dummies. And that allows us when you do least squares, you get the group-specific means. Cool. If you do that in a tree, you're taking a fundamentally discrete set, you know, the list of all the umpires, and you are peeling one and putting it on the right and keeping everybody on the left. And then you're taking another one, and so you're partitioning this large discrete set by removing one element at a time. And so what's happening is that whenever the tree split on umpire, it would take all of the data for that umpire and put it on one side of the tree and everybody else's data on the other. So you're not actually borrowing strength across umpires in any useful way. It's not the, oh, there's this group of umpires where this thing is happening and that group where something else is happening. And so it was just a, it wasn't really a statistical thing. It was sort of a computational or a heuristic thing of how do you represent these data?

So we rebuilt Barker, or rather I spent four years just rebuilding the entirety of that code base. So it would do one very specific thing, which is if it sees a categorical predictor, rather than one-hunting it and then doing continuous decision rules for those, just randomly put some elements on the left and some elements on the right. And see which ones perform better. The same kind of algorithm that you, yeah, yeah. Almost. But then, of course, the referee came in and said, but you have to, how could you possibly find an optimal split? But, you know, if you think about a cart thing, this is a combinatorial problem. It turns out there's a paper by Fisher in the 50s, not RA, but a different Fisher, which solves this for cart splits with categorical predictors. You can sort your categories by the average outcome, and that turns this combinatorial problem into a linear problem. Yeah. But in part, you don't even have to do that.

You are proposing random splits. So, like, this is the other thing. It took a long time for me to realize the whole algorithm just does random splits. It doesn't do informative splits, or you don't need to do informative splits. Let me put it that way, which means the trees are actually individually really bad. So I'm not actually finding good splits of the empires or the things. They're just kind of stumbling around in the dark and self-correcting because there's a boosting. and the analogy is to boosting and this kind of sequential updating one tree, fixing everybody else. So you're sort of training a tree to predict the residual of everybody else's. So it took me a long time to really understand the algorithm and why it was so easy to just specify a prior. How do you partition a discrete set? You sweep over it and you flip a coin. You don't think, you don't try to find an optimal split. You just 10 of you over here, five of you over here. does it get accepted or not?

And so it becomes a very simple algorithm and a launching pad for how do we do so many other cool things with BART. So we, yeah. Now, does the default, though, then, is it on average always going to be like half and half, or is it sometimes going to like it'll get into, like if there were, so like suppose there were five umps, or not umps, you said the catchers are the ones that are different. Like, you really want it to be 5 and 20 or whatever. Can it get through that? No. No. So this was another thing that, like, as part of this, we did a lot of experiments. And with one of my close collaborators, we did the following thing. We divided with, you know, 100 observations, and we split them into two groups. And we said for group 1, plus 5, group 2, minus 5, and very little error. So this is an easy regression problem. and all we passed part was kind of the idea of the observation. The idea was the first 50 would go plus 5, the next 50 would go minus 5. And we checked, did any of the trees get the groups correct?

No. None of them got the right group. But the predictions were perfect. Yeah but the predictions were perfect because that first tree takes a random split and then the second tree comes along and is like well I got to fit the residual and that first tree did something weird so let me kind of correct it again Jigger it Yeah And that was this kind of amazing thing, like, what is this algorithm doing? And what is this method doing? And so I just got real obsessed with it. Yeah, that's so cool. Can it work if you I guess maybe we would just pick the people. Like, if it was an imbalanced group What happens when it's an imbalanced group? It just. It comes out in the predictions. Yeah. Because it will, like, put, it'll always put those five on the left or whatever. And then the other 15 are sort of maybe waffling between. But then all the times when the five are on the left, it's better. So then the predictions are going to come back.

Cool. This is so cool. It was just one of those things where, like, I just wanted to run BART on my baseball thing and call it a day. and then it turned into writing at least three or four papers and this like brand new r package because in order to fit these new types of models and more flexible trees you have to change the back end lots of new data structures a lot of new algorithmic decisions and then you have to expose all of this new information to kind of the underlying low-level code and it was getting to the point that my collaborators were saying okay so you have nested variables and you will like automatically detect which ones are nested and how they're nested. And was this categorical? Is this continuous? Is this kind of a discrete continuous variable, like an ordinal variable? All of those things. And I said, yeah, you've got to rescale it and you've got to convert factors to integers. It's really, really easy. So I was the only one able to use it. And my collaborator said, this is absurd. So we spent two and a half years building a front end.

So now we have a 4-mil interface. So I'll put it here in the chat. But you could really write something like part of X1 plus part of X1 plus X2, stuff like that in R. That's cool. It'll catch all of this stuff. And so it's only – it's soon ready for the big time. We'll announce it formally. But it's been on cram for a couple of months. But, yeah, this all started because I was messing around with baseball. That is amazing. this is so awesome well i'm excited this will be um so so for our listeners we'll put some of we'll have you put some of our little we have a little blog so that people can sort of see how it works this is so cool i feel like in the way that it's set up to particularly talking about like like more of the like in the context between like the umpires and the catchers right that's the it almost is reminiscent of the love island like the didactic the didactic interactions i feel like

where like these people are paired right and you have repeated observations of like what happens when they are paired um and it's almost i guess in that case it would be partitioning like how much of the variability in i don't know what what how you would measure the outcome i and i don't know if you could do that with like a binary rather than a numeric but it's like how much variability and whether or not something is a strike is describable essentially by like synergy or lack of synergy between the umpire and the catcher, right? I'm so glad you asked this because this is a project that we're doing right now. Cool. And that is, this project is one where the sports and stuff community is amazing. And I just got back from one of the premier conferences over at CMU. and one of the people there, I've known them for years, and we decided to start working together on this thing because baseball changed their rules about what is a ball and what is a strike.

And they're moving to this system where it's kind of automated, and it is a thing where it takes some of the variability umpired to umpired out of the game because they're now being asked to call a very consistent strike zone. So we asked, how is the variability changing year over year? and I don't I am not nearly technically competent enough to share this on a screen but I can send this to you we made a gif that looked oh since 2009 all the way to 2025 every season we looked at at every location in that plan of home plate we estimated the probability that each umpire would call it a strike I took the standard deviation so like the umpire variability and then we made a little bit of it so you could see that sort of donut shrinking, moving, and kind of like moving around in space. Where is the variability in these calls, like in a spatial thing?

And of course, to make it somewhat causal, we want to know what is the causal effect of this new system. And so we have, this is one of those challenging questions because there's no control. Everybody has this new policy, new technology, new rule. But then you have this idea, we have this time series at every point. What is the probability of a call strike or the variability in that probability across umpires? So we estimated those with a part, and then we slapped a Gaussian process to kind of finish that year-over-year trajectory, see where it was going, compare it to what we see in 2016 or 2026, and then we'll get a very precise number of did this system make umpires more or less variable? So I'll send that elsewhere. It's sort of a fun little, like a fun little kind of artifact. Yeah, that's awesome. So, okay, I'm curious. I watch baseball recreation, but I don't watch, like, super regularly.

But my husband and I were just watching the Guardians game last, maybe it was last week. I feel like they were playing the White Sox, but maybe it was the Raiders. I can't remember. We were watching a Guardians game. and um and it there was a bad call like a clearly bad call like the the um i can't even remember what happened i remember my husband i don't think i was watching when the initial bad call happened uh so anyways the batter like should not have been able to bat again but they were and then they you know they show you the box on the screen the batter hits the ball it's it is in the box But there had just been this bad call, and the ump calls it out. I guess he must not have swung. The ump calls it out, whatever. And then the batter just, like, leaves and didn't dispute it. And Patrick was like, oh, it's a makeup call. But from what you're saying, is that not possible? It is. I think I should just see it happen. Okay. I feel like I saw it, but I don't. So it's a system where you get to challenge two calls per game. Got it.

Okay. And if you win your challenge, you keep it. So you have two to lose. And if you think there was a bad call, you're supposed to tap your head. So it's a very, like, far be it for me to say that baseball does a lot of things right. They rolled this out correctly, and it was perfect. Because in the first week with this new rule, you'd see this umpire make a call, and the catcher immediately tap his thing, and they'd go to the screen, and you'd see the graphic of the pitch coming in, and it would be very precise. And this catcher did it to the Empire like three times in a row of just like going in there and like challenging it and kind of saying, I know what I'm doing and you don't. So it's been a fun thing. It's fascinating. I mean, so if you challenge and it was actually wrong, you get to like – You keep that. But if you challenge and it's like, and the ump was right, or the caller, whatever, do you get to, what happens?

I mean, you live with the result. Yeah. And then you lose the chances. Yep. Yeah. Because Patrick was trying to tell me, he watches a lot more baseball than I do, and he was saying that, like, it does seem to be shaking out that there are some that challenge more often than others. Like there's some, and there's like a strategy to sort of like when you're going to challenge it. Like this was one where like they could have challenged it, but there was such a bad call just before that it was like not, it felt like there was sort of this like, you know, I don't know, for whatever reason everyone decided like let's just move on from this because it was, you know. So, yeah, that's very intriguing. I wonder if you're like for a follow, maybe you've already done this, but a follow-up analysis of like the probability of challenging. Because now it's like now maybe it's not on the ump anymore. It's on the players. But there is still, right, there's still like variation. I would assume. There is. There was some work that I just saw at this conference by some awesome students. They're actually high school students who are presenting this work. And they talked about how sure do you have to be in, like, your ability and, like, what actually happened to make it worth your while to challenge.

So it's a very cool decision science problem. What do you think happened? How sure are you? So there's going to be a ton of papers. And we're already seeing it, like, scrolling through the archives, and there's a lot of ABS papers out there. And so it's fun. It's like the new thing everybody wants to do. But there, oh, my gosh, there's so many categorical predictors. It's individual people. It's space. There's time involved. It's complicated modeling. I would imagine. Well, like, this one, where it's like the temporal, the piece that there was a bad call that just happened. Like, how would you represent that somehow in the data? Because I could see it with my eyeballs that that's what just happened. But it'd be probably kind of hard to, like, pick up the signal of, like, there was just a bad call, so this is going to be a makeup, and I'm not going to challenge it. Because, you know, that kind of, like, and I'm sure that happens, you know, occasionally at least. Absolutely. Well, this is so cool. Okay, well, I'm cognizant of the time. Sarah, do you have any other follow-up questions before I?

I have one follow-up question as a person who would admittedly have floundered in a sports intro stat class. Sure. Are there other sports where you are constrained in the number of, like, I'm thinking like football, like American football. Yeah. They can also call for, like, the play to be reviewed or something like that. They can essentially complain in a formal way about the way something went. But are they restricted in how many of those they're allowed, or is baseball unique in this? No, this is common across most sports. Anytime you have the ability to challenge. So in football, if you lose a challenge, you lose one of your timeouts. And then there's like a certain number. I think it's two within certain points of the game. There are other parts of the game where you can't initiate any challenges. In basketball and in college basketball, too, there's challenges. and the coaches only have a certain number.

In cricket, you can challenge calls, but only three, depending on the format. And they're all the same dynamics, right? If you're successful, you keep the resource. If you aren't successful, it burns and you have this consequence. That's very interesting. I think it just happens at the game that Matt and I were at this weekend. because I think it was in overtime. I actually wasn't there by the end of it, so we went home with our kids. But it went into overtime, and it was the Syracuse-UNC game. Syracuse supposedly caught in the end zone, but it was debated whether they did. I actually don't know if UNC formally challenged, but from what Patrick described, it was like there wasn't actually a camera down the line. For whatever reason, there wasn't one that captured that exact. So they couldn't challenge the play because it's just like none of the cameras happened to have captured exactly what was needed to have been captured, which is also so interesting to me because I keep hearing about, you know, as you said,

we've got 360, we've got all these points on everybody where they're moving. We have all these cameras that are doing everything. And then it's like, oh, but really, like in the overtime when someone like catches in not the end zone, we like not one camera caught this. So this is another thing that the tracking data in football, at least in the league, the players used to wear chips in their shoulder pads. And so that resolves player position sort of well. Yeah. They tried with the ball, and it was not nearly accurate enough. But now they're, like, moving on to Hawkeye. So if you watch tennis, you know, the little ball-out camera system? they're installing that in all of the in all the arenas and all of the stadiums and it's not coming to college yet but give it give it a couple of years um that's what they use in basketball already right or no is that uh they have that in basketball um but they're not giving us access to the data so if there's anybody in the nba listening to this like we're gonna have words my student samir

my student i have a student that was he was like my undergrad thesis student he works for the nba And that's the data that he works at because I've talked to him about it before. And he was like, he was like, there's so many. Like, it's just like too much information because they have like all for every player. He's like, they don't even really their team. I think he's on the data science team for I can't remember what exactly the subdivision is. But it's like they're the ones that are supposed to be doing something with this like onslaught of information. And nobody is like quite yet prepared because the methods just haven't caught up. I mean, it's something like a million and a half records per game. like yeah well they end up like what they end up averaging down to like where you sort of might as well not even have this really nice granular data but you know yeah it's interesting cool okay well this was awesome oh i forgot in the last one to do yeah thank you for coming would you like to so we do this like you can rank your analysis on a scale of one to five fetches or no actually one to any number of fetches.

Would you care to? Why are we trying to make it happen? Nine fetches for nine innings. Fair enough. Yeah, that sounds good to me. Nine fetches for nine innings. I love it. We've gotten too cute with this there. This data is going to be so messy. Like if we ever tried to pull like everybody's fetch ratings. Because I think literally no one would be shocked that we skewed up for this. Could I go for it? If it's on the table, I'll take however many calories are in a calteen bar. That's hilarious. Amazing. Yes, that is on the table, and you have successfully done it. Yes, this is excellent. Well, thank you for sharing with us. You go, Coco. We're crushing to the V-Girls. We're mean as an average, but actually very nice. Tune in next time. We'll see you next time.