Vibe Coding's Uncanny Valley with Alexandre Pesant - #752

I'd like to thank our friends at Capital One for sponsoring today's episode. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade-in value. Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One. all right everyone welcome to another episode of the twiML ai podcast i am your host sam charrington today i'm joined by alex bazan alex is an ai lead at lovable before we get going be sure to take a moment to hit that subscribe button wherever you're listening to today's show Alex, welcome to the podcast. Yeah, thank you so much for having me. I'm super excited to jump into our chat. We're going to be talking about all things VibeCoding. To get us started, I'd love to have you share a little bit about your background. Yeah, so I've been at Lovable since July last year. I was one of the first engineers. That was a little bit before things took off and, you know, it became what is today. It's been super fun building this. I believed in a mission from the start of letting anybody write code without knowing how to code. I believe a lot in this. And I joined Lovable because I actually worked on an agent that topped Sweebench for a day or two. It was called Delvin. And that was on the GPG 4.0 times. And the scores at the time, the high scores were 23, 24%. And now we are at 80% plus on Sweebench Lite. it's pretty crazy how things have gone, really. And yeah, so I've been working mostly as software engineer and then product manager over the past 15 years. I am a failed ML student. I actually studied machine learning 15 years ago, and I followed the online courses with Andrew Eng, you know, the OG Coursera. Everybody who did that, raise your hand. Yeah, I've done it even a couple of times over the years after that because I worked as a software engineer, so I was not touching really ML. And at the time, I thought it was too late for me to do machine learning. I thought it was too late, and I was writing neural networks in MATLAB on my computer, you know, and it was before GPUs were a thing. Right at the time, it started to be a thing for machine learning, and I have regrets there that I didn't pursue this, you know, with higher conviction. And I've kept playing with these things on the side a lot over the years, you know, building small models. And I'm super happy that I found my way back around this from the application layer. But I just, I love working with this technology. It's just incredible. And I think this is kind of just the beginning, you know, the exponential is real. Yeah, I wonder sometimes if, you know, having that experience in the space versus coming into it fresh is, you know, a bit of a handicap versus an advantage in a sense of I often find myself processing things through a lens of having been in the space for, you know, a dozen years, whatever. And it's moving so quickly. Sometimes I feel like that doesn't serve me. It's like if I could just look at this without all that baggage. I fully agree. I see some younger people, you know, they are 18 or 19. We have some of these cracked engineers at Lovable, as we call them. And they don't understand a lot of things, but sometimes it's for the better. They just know, you know, what the tools of today can do. They are very creative in using AI, even more so than most of us. I mean, we're still pretty tech savvy. Obviously, we're not, you know, really like behind, but they just, they don't care. this is just how things work. And going forward, I think this is going to be even more amplified and even more, which is a bit scary. As you get older, you want to stay in touch with how things work. And yeah, I think there are quite a bit of people who have not maybe liked LLM as much as they should because it kind of took away a lot of the craft of ML because you could kind of brute force your way through a lot of things now with LLMs. but you know I'm just excited about moving to the next layer of abstraction on top and figure out okay what can we do with this crazy technology yeah yeah yeah you know the clear thing for us to talk about is vibe coding which you know in some ways feels a bit kind of flash you know I don't know trendy and the kind of the analyst in me wants to kind of set back and take apart like AI-assisted coding in all of its glory. And so, you know, maybe I'll lean a little bit on that and have you, you know, start off by talking about how you think about, you know, where Lovable sits, presumably Vibe Coding, in this broader AI-assisted coding sphere or ecosystem. Yeah, I mean, I think it's pretty fun to sometimes look back at the timelines of the AI-assisted coding, as you said, where I think the first big product hit, I guess, was Copilot, really, for a lot of people. I actually don't remember the year of the... Or even GPT-2, right? Let's write this webpage with GPT-3. Yeah, GPT-2 was still kind of pretty quickly nonsensical, but GPT-3 was like, wow, this is starting to look like something. thing. And then Copilot for developers really was the first thing where you realized, hey, this is kind of useful. And I think it would be interesting to go back in the past and look at comments online from people with the timestamps of where they're saying this was never going to scale or get better. But I think a lot of people are just feeling that there was no reason it would stop at completing the next line and then it could do two lines and three lines and four lines. And then obviously ChatGPT came out with GPT 3.5 and I remember the night it came out, I was working at a medtech company here in Stockholm and I couldn't stop using it for coding with me. I was just testing it. I was working on some plugins for some UI portal we had and I was just asking things, hey, can you try this? Can you try that? And I was kind of copy-pasting And I think in hindsight, it was probably terrible, extremely terrible. But I was, you know, I was shocked at how good things were. And where I'm going with this is that, you know, at the time, I think it felt, at least for me, it felt pretty easy to see where this was going. And the future was that we would stop writing code as we do it now. We would not be typing the characters as we are now. And we would move on to higher level abstractions. And, you know, going to the vibe coding term coined by Andrei Kaprafi, I think, you know, I remember his tweet saying the hottest programming language is English now. And, you know, I think that's something we could see coming at Lovable. And so Anton and Fabian, the founders, they created GPT Engineer. You know, that was one of the largest, most popular open source projects that came right after GPT-4, I think. And I think at this moment, like so many of us really knew, okay, this is happening. This is really happening. Things are going to get kind of crazy. and I think they have gotten kind of crazy. And whenever I hear people saying that the progress is not as fast as it used to be, I think, you know, the timelines are just being distorted because if you look at the calendar, it's actually kind of crazy how things are moving and we are also jaded by the progress and we expect it to be, you know, normal. But it's really incredible. And so the perspective where we come from is that our belief is that, you know, this is working, this is going to work. You are going to be able to write code without knowing how to code at all. And I think I have a hard time understanding why some think this is not a legit way of thinking, because, you know, when you write codes, you're not writing machine code, you're writing some other abstraction on top of it. And that code gets compiled. And at no point you ever look at it. So what is the difference between Vibe coding and Vibe compiling? You know, it's the same thing. When you compile your code, it is translated into something else. And when you're Vibe coding and programming in English is translated into something else. And it's conceptually the same thing from the perspective that the language that you're coding in is not the language that the computer cares about. But I think the thing that maybe people are responding to is injecting this very statistical element into the middle of the mix, which makes things, which causes problems, right? It does, but I think, at least the way I see it, is that it's a transition phase where things are, you know, it's like the uncanny valley of programming where things are starting to look, you know, like it's happened for image generation, for example, where things started to look very real, but kind of not very real. And now it feels like we're way past this. It looks perfectly real. Same with video now, since a couple of weeks back. And programming is going through the same change. and I just find it hard to believe that it's not going to work at larger scale. Obviously, it's a bit like self-driving cars. You know, there's a long tail of things that are not going to work straight away. But Lovable, as it is today, can already build pretty crazy apps, to be honest. I've been working here for more than a year and I've been, you know, using the product all the time and I often rebuild the same apps over and over as a way to, you know, test things and do my vibe checks, speaking of. And, you know, sometimes nowadays when I build this, I kind of run out of idea for my little app I'm building because it's kind of, it's just working. It's doing the thing I want. And it's really getting to a point where it's just, it's really impressive. Like I'm not trying to sell our product or anything. Like, I mean, vibe coding in general is getting super impressive. And if you use coding assistance like Cloud Code or Codex, it's also getting pretty, pretty crazy. So I see no signs of stopping and the labs are investing everything they have in there. So, you know, I'm pretty confident. Yeah, I should be clear that I'm a believer in, you know, both Vibe coding and AI assisted coding. You know, having used a variety of different tools like, you know, there is, you know, they certainly work to a degree now, depending on what you want to do. I think, you know, where I hear people reacting kind of negatively or pushing back, I think one, it's like, you know, let's set aside a certain amount of cope. You know, I'm a software engineer and like this thing is threatening my livelihood. So it doesn't work poo poo. Like, let's set that aside. I think there's a legitimate pushback to software engineers. Software engineers are cooked. That whole meme, don't study software engineering. It's going to be written all by code with the implication that it's going to happen tomorrow or today. Or we're there already. Like I think some people, you know, that's something that I would push back on, right? Like we're not there. Like, you know, we need software engineers. We need, you know, skilled people to apply these tools. And I think that comes off as like poo-pooing them when really it's more like pushing back on, you know, bombast. I think I totally agree with that perspective. by the way, we're hiring software engineers at Lovable. You know, we're not pretending we can be replaced. And it's the same thing, you know, like Dario Amadei from Anthropic was saying, you know, maybe you've seen this online six months ago. AI is going to write 90% of the code in six months. And they're hiring tons of engineers. But I don't think these statements are contradictory at all, to be honest. I think that you get so much more leverage now. There's a lot more code and we need a lot more engineers to wade through all this code that the AI is generating. Yeah, so maybe there's this angle of... That's one way to think about it. You need to hire people to fix the Vibe-coded code, which is maybe a part of it, to be honest. But I think that this is more that you can just do so much more now and there's much more to do. So, you know, the level of ambition humans have will never go down. So whatever the tools you have are, you will need more people to just help to get more things done. even when you have very powerful AIs. And so I don't think that's too much of a contradiction. They're saying that you're hiring software engineers, yet you believe that AI can write most of the code. Like I think for a lot of the tasks we do, we don't write the code, but we still need to think about it. And we need to think about, you know, the user experience. What is the thing we're trying to build? What is the actual problem? And you need, at the moment, at least you still need humans to have some sort of desire to accomplish something and, you know, to build a product based on some data plus some emotions and, you know, and a vision. And we need more of these people that are lovable because you can build things so quick. So you want to even have even more people to test things faster. So it's like a spiral, you know, it's accelerating all the time. And the code that is written, the characters are not typed by us to a large extent. That I think is true as well. Yeah, I agree with that. And I think about it through a couple of lenses. One, through the enterprise lens, like if you think about the software that exists to help run the typical enterprise, like the amount of software that exists there isn't fixed at whatever quantity it is because all the problems are solved. it's because like there was a very strict prioritization because they couldn't possibly have all the software that they would have wanted to automate all the processes and help make all the decisions etc um you know so they you know build what's most important um you know through you know large projects they buy a bunch of other stuff and a bunch of needs end up unmet and you know, part of the promise, I think of, you know, again, whether we call it vibe coding, assisted coding, agentic coding, whatever, is that, you know, now we can kind of take off a much larger bite of kind of these problems and, you know, automate them. And I think the same thing is really true in an interesting way on kind of the personal slash consumer perspective, you know, this idea of like the single purpose application or the single consumer, single customer, single user application. Like I've got a need. It's not a need that's big enough for, you know, someone to try to solve as a, you know, via a company. But like now these tools allow me to just solve the need that I have in a way that, you know, works specifically for me. Like I think those are both, you know, really powerful, you know, indications that there's a lot of opportunity here. Yeah, this is here to stay. I think there's no, I don't think there's a bubble. There's a lot of hype cycles, obviously. But I think the value created is real. And we have lots of users to prove it. Lots of happy users, sometimes starting businesses. So they're trying to create software for others to use. Lots of people building what you call this before, like personal software. I've seen some incredible projects where, you know, the number of users is maximum five. It's a family, like a household where some people have built their entire household management systems to a degree that it's honestly unbelievable. Like they've gone further than anybody has ever thought about the problem. And it's just for their single home. You know, I find this incredible and so cool. And it creates value for them. You know, they're getting value out of it. It makes their lives better. Some others are, you know, using it for raising money. So they're building prototypes and they're going very quickly, validating ideas, building the MVPs, and then, you know, trying to raise money. This happened a lot with lovable products as well. Others use it just for designing things and get a feel for, you know, prototyping. There's just so many use cases. It's a little bit like ChatGPT in the sense that it's very, very general. And there are people who use it also as just for having fun, you know, just building things like gaming a little bit. I also think it's pretty cool. So there's a lot of real value being created. And that's here to stay for sure. So I don't think we ever really bit off kind of taxonomizing the AI coding ecosystem. Yeah I think it you know is it fair to say that lovable is going after you know you know what I think it was like pure play vibe coding, meaning, you know, I'm an individual or like maybe a small team, a small startup or something like that. And I'm like, you know, I've got this idea for a thing. I'm starting from pretty much a blank slate. I need to incorporate kind of front end and back end. And I think those things all, you know, those are some of the things that to me characterize maybe like a more principled view of like what vibe coding is, you know, relative to, you know, AI assisted coding or other types of AI assisted coding, because there's, you know, AI assisted coding would be the umbrella, then agentic coding would be the umbrella under that, then vibe coding would be under that. And I say all that, and then, you know, part of me says, like, you know, as always, dude, you're trying to make too much meaning in here, and it's all the same thing, and people are throwing these terms around willy-nilly, and, like, you know, the distinction, it's a distinction without a difference for a lot of, in a lot of conversations, at least. Yeah. No, I do think you're right, though. When you hear vibe coding, you probably think a lot more about creating new projects from scratch rather than working on an existing code base, for example, where even though, like, you know, we talk about these things internally, we can say, oh, this was Vibe coded, even though we work with an existing code base. But I think you're right in your general perception. Yeah, at the end of the day, it's all the same thing. But maybe Vibe coding right now is on a smaller scale, I guess, smaller scale of projects, right? And there's a whole landscape of different companies and products attacking the problem through a different angle. That's for sure. And so Lovable is starting to, from the, you know, empowering the 99%, that's our mission. The 99% of people who cannot code. And of course, this has a very consumer-oriented, you know, connotation there. And this is where we're starting from. We want to empower all of those who can't code to build things. But if you think about it, it actually does not really mean consumer. It means, you know, even in enterprise, there's a lot of people who cannot code or are not developers. And we have lots of users, for example, designers and PMs building prototypes internally and, you know, like speeding up the communication process with engineers or even stakeholders just saying, hey, we could do it like this. Look at it. You know, it actually works like just try it out. which I think, I mean, if you worked in any software company that's sufficiently large, you understand how much communication and overhead there is and things getting lost in translation and the frustration from a non-technical stakeholder to try to sell something, you know, to developers or even product managers where you can kind of just... Please, just as one feature, it'll change everything. Yes, yeah. And now you can kind of just say, hey, this could work like this. I even tried 10 different versions of it because it took me an hour. You know, it doesn't cost much. So the vision, I think, of Lovable goes probably a lot deeper than we think in the start because we really want to power the businesses of tomorrow as well. You know, we talk a lot about the first solo corn. And I think this company might be built on top of Lovable because this is really what we're trying to achieve. like bring everything under one place where you can start your company and build a company if you want to and be successful. And I think the incentives of individual users and small businesses and enterprise are the same at the end of the day. Just trying to build great products and move as fast as possible. And VibeCoding products are, you know, they're helping you getting there. So we have potential users really everywhere. And this is just a start for us, obviously, in this journey. We're a small company, a new company. We have a lot to learn. And we want to learn what everybody needs for their use cases because obviously enterprise and consumers are different. To some degree, we're talking about this idea of democratization of software or of code, like it's a new thing, but it's not at all a new thing. And like, we've got, I don't know, a decade or more of kind of low code, no code, you know, as a prior to all of this five coding stuff. And, you know, one thing that I'm finding interesting, you know, when we looked at the recent OpenAI, you know, agent tool, you know, we're starting to incorporate some of these like visual GUI, you know, low-code, no-code-esque ideas into the idea of, you know, building agents or building things using AI. Like, I think the question in here is, you know, given that you're going after this 99%, like, is code the right thing that, you know, the right kind of, you know, medium of communication or currency or something? or do you envision like some higher level yet abstraction that, you know, whether it's visual or something else, like, you know, is there something between English and code or is there a visual thing that sits alongside that is going to either be required or be an accelerator to, you know, enable folks to really build things? Yeah, I was a little bit surprised by the OpenAI agent builder because it's very, as you said, low-code, like a lot of dragging around and, you know, manual things, which I don't know, you would expect something where you can just, you know, chat and say, hey, this is what I want, I think. And I mean, the whole reason VibeCoding products are so cool and powerful is because they write code because you can do anything with code. There is no, you know, limitations and there is no built-in guardrails, which sound good for protecting yourself from doing something that's not working. But I think basically my stance is that models are getting so good that you really want to remove the guardrails. You want to set the models free as much as possible for a lot of scenarios. So I was a bit surprised that they went for this approach. When it comes to what's the right medium for communicating to users, I think you're right that when, I think especially when things do not work, Because when things work, you know, when you ask for something and it works, like, why would you want to see, why would you want to look at the code or any diagram or anything, right? It's working. So you're going to focus on something else. But when things maybe are not working and you need help, then looking at code is definitely not an option for a lot of our users. And even for an engineer, if you're, you know, if you just open a code base that you don't know, it's going to take you a long time to get familiar with it. Like I've done this a lot with some of my own lovable projects and I clone them and I look at them and I don't know how it works. I've never looked at it, even though I know the projects very well. So it's the same thing. And so I think we want to keep code as this sort of very strong way of doing whatever you're asking for. But there is definitely a feeling that the UX of the future is still not really there today, where we're going to find ways of being more, a lot more visual and interactive and try to always surface to you whatever makes the most sense for you to understand how things work or even, you know, how to reason about what you should do next. Because like, you know, vibe coding is basically product management, essentially. It's just, you know, thinking about, okay, sequencing, you know, what should I start with? What should I do next? How can I validate my idea? If you're really serious about, you know, building an app for users, when should I publish it? What should I do in the product to keep our users? You know, you start thinking about retention loops or, you know, the sort of higher level concepts. So if you're serious about this, you probably need different ways to visualize it than just text, text, text, or code. And I'm very excited for this. And I think we will try to have a really nice take on this that feels more AI native than just, you know, something like agent builder. But yeah, I agree with you. It's going to be very interesting. And again, you know, with great power comes great responsibility. And when things are more powerful, they're more likely to go wrong as well. You know, and I think that's probably why OpenEye went the way they did. Yeah, there's a couple of directions I want to go in this conversation. And I'm mostly putting this out there as a little bit of a bookmark. One is like, you know, how lovable works, what you needed to do behind the scenes to get all this going. But I think folks will also find a lot of value in like your assessment of the, you know, state of the art from a vibe coders perspective in terms of what's working today and what's not working today. And I think I want to start there because it's a bit of a natural segue from what we're just talking about. um you know just as the tools and models are evolving quickly that you know is evolving you know super quickly too like um it's it's it's fine i was just thinking about this the other day because i was playing around with uh i think the new gemini um gemini enterprise has its own kind of take on the agent builder thing. And one of the prompts was like, you are a X. And I'm like, that always feels like dated to me. The like, the personification in a prompt. And it, it kind of makes me think about like how, you know, quickly, you know, when I think about the evolution of vibe coding, like, you know, it was first like, okay, we've got a text box and we're going to type in this thing that we want to see and we're going to hit a button. And we're just going to keep talking to this thing and telling it like, you know, what we want to see next. Right. Kind of like a like a slot machine of code. And then I think, you know, as a community, we got a lot more sophisticated about it. And like we'd like make PRDs and turn those into task lists. And like we'd have the, you know, the agent iterate through that and, you know, maybe do some testing. and that's mostly where I've gotten to. I want to know what's next also. You know, I want to know what's beyond, what are you seeing beyond that? I also, you know, there's a little bit of dissonance and I think this goes back to the conversation that we had about like people poo-pooing it. Like, I'm always trying to like, you know, wrap my head around like the impression you get with folks. I just like type this in and it's all, you know, it all works, right? And that's not my experience. Like, you know, a particular point is like, I find that I personally have much better results when I actually look at the code that, you know, is returned to me, reason about it, have kind of a broad kind of architectural direction, you know, and kind of push the model, the agent in that direction, as opposed to the slot machine. But, you know, on social media, like, yeah, nobody, I don't ever look at the code, blah, blah, blah. Like I've got my, you know, I guess the meme is like I've got my 50K a month ARR app and I've never looked at the code once. And so that's really a long way of saying like, you know, where do you think like the kind of cutting edge techniques are? Like what's BS? Like what really works? Like how do you approach it? You know, all of that. So the models we have right now are extremely powerful. And I would say they're extremely smart in the sense that they know what to do next, given some new piece of information. So if you think about agents being essentially a loop of, you know, tool calling, like just to make it simple, tool calling that you get some results from your environment and then you take the next action. One thing that's changed a lot since my days of building my first agent a couple of years back is that agents are excellent at knowing what to do next. I would say that, you know, they're probably better than humans for a lot of things deciding when, you know, oh, now I should probably search the web because of this result. I should do it right now and not before. Oh, I should maybe take a step back. You know, they're starting to get really, really good at this stuff. And quite often when we fail the models is because we're not giving them the right feedback and the right pieces of context. You know, we talk about context engineering now. It's the new way of saying prompt engineering. But essentially, when you have an agent, you're just making API calls to an LLM provider, right? If you zoom in, it's just a loop of API calls. That's it. And for each API call, you have a certain amount of context. You can stuff things in. And I do think that no one out there is even close to mastering this at the moment. You know, there's still so much that can be done to think on every single iteration. What could we add? What do we know could be useful at this moment to just guide and nudge the model in a better direction? Because the models, given the signals, they will really be excellent at finding the next step. And I think this is where we have a lot of work to do and everybody has a lot of work to do. And that, even if models were to stop improving, I think products like Lovable would still improve a lot over the next year or two, because there's just so much stuff that's just not done and invested in, because there's just so much going on, so much low-hanging fruits everywhere. And I often hear people in the field of proper ML and LLM research saying the same thing, that it's low-hanging fruits everywhere. We feel the same. but then the one area is that where the models are not good at enough at the moment still I think is software engineering which sounds a little bit weird to say but they are really good at writing codes but they're not really good yet at thinking about a code base you know trying trying to reason like what you were saying before you have to look at the code and you're you're trying to apply your your skills and say hey maybe we should structure things differently and do things differently I think this is where models are still not amazing, but also the agents, the frameworks and the agents out there are also not really good at incorporating this into the flow. You know, if we talk about agents being just a simple loop of tool calling, maybe this is wrong. You know, this is not the right way of thinking about it at the moment. you see if you use cloud code for example you see all the sub agents you can invoke and you're starting to recreate this sort of a hierarchy of you know different concerns like my concern is low level code and my concern is the architecture and you know i don't really like this sort of personas where you start recreating artificial boundaries like with humans because you know i mean lms are better than this they should be better than us and you know be able to be a great product manager and coder at the same time. But that's something that's still pretty hard at the moment, I think. And where this is going next, I think is more, like on the application layer side is asking this question. You know that, so I believe that Vibe Coding is working and is going to work really, really well. So when it works, what's next, you know, for products that are building apps with you when everybody can build an app, you know? So what do you need as someone trying to build something? How can we make the friction go away? Because there's so much friction that has nothing to do with the models in general. You know, coming back to the UX of it as well, I don't think we found yet the correct UX. And I'm excited for that. And using LLMs everywhere, of course, and agents, and not just in a cogeneration pipeline, but an entire journey for an app lovable. I think things are going to look very different in a year than they do now. All that said, are there practices that you found that differentiate the folks that have a lot of success in building, you know, reasonably complex, you know, above average complexity applications with a tool like Lovable relative to folks that, you know, that fail or aren't able to get to the finish line with their projects? Yeah, absolutely. I think one way to be successful is to know what you want. So first you have to know what you want, I think, which implies that you need to be good at communicating with the model the same way you would have to be good at communicating with a coworker. you know I talk about this a lot with with users like the expectations some some users can have on a model to understand what they mean are sometimes extremely high and you know if you were to show the same conversation to another human they would not understand what you mean when you ask for something in five words you know they would be they would just be asking wait I don't I don't understand this is very ambiguous what what are you can you describe this a little bit more and then I think we have a lot less patience with AI in general sometimes when we assume that they know everything we know and understand everything we mean. So I think very powerful users of Lovable are using, we call that chat mode. So it's a mode where you can chat and plan a little bit deeper. So you take your time to prepare what you going to do You make sure that you on the same page you and the model basically And then you can go into the implementation phase where things are a lot more specified So it what you were saying before about PRDs essentially that you want to think about communication and thinking about things before. The second thing, going back to more what I was saying about product managers, is to try to think in terms of sequencing, you know, what needs to come before what. So it's like sequencing slash software design at the same time. You know, how can I build something solid based on good foundations? Typically, when you build an app that's going to require authentication and users to log in, you probably should fix this from the start because it's better to have it in place forever than trying to shoehorn it, you know, a little bit later. And so, you know, even though it's not software engineering, I think Vibe Coding on its own is a skill as well. and you know some of our users they can literally build I mean almost anything because they know how to slice it you know and how to check things and and another thing that makes people very successful is to stop the bleeding when things do not work and I think it's true for any type of AI assisted coding or even a conversation with an assistant like chat GPT if you know if things are not going right and you have a misunderstanding you should not try to you have to know when to pull that plug Yeah, you should just go back, go back, you know, revert or go back, edit your previous message and try to communicate a bit differently there rather than trying to get out of a rabbit hole. I think that's one thing that makes a difference between, you know, failing and getting stuck and never getting out of it and someone being successful. And it's really, really hard to give up on the thing you're, you know, it's a certain cost, right? So taking that to the extreme, you know, there's a whole, you know, meme, I guess, about like the way to use these tools is to, you know, like each time you issue a prompt, like you're testing that prompt. And then if it fails, like you start back at the beginning and like you issue all of your, you know, known good prompts or you have them in a doc and then you have it like go through and build up from nothing, you know, or build up from some checkpoint based on like what you know has worked well and got you to the to this point. But like, it strikes me that that's not necessarily quite the same as like, you know, bailing on a task and popping back up and starting from scratch. Like, do you see a lot of folks, you know, is that distinction one that you see and do you think it's a useful technique? I know some users often build a first version of their app as a way to explore. And then they rebuild it with essentially all the requirements well-defined a second time, like 10 times faster. And I've done this myself with some apps I've built when I was kind of playing around, not really knowing where I was going. You know, so I built something and then I realized that I don't really want it to work like this. So I kind of take a lot of turns and it takes me a while to go from A to B. and then I can kind of rebuild it from scratch when I know exactly how it should work. And it is actually pretty good. And you know that models are really good at giving a proper plan, you know, to execute well. So when you can give them a more holistic view of what you're trying to build, you're more likely to be more successful. So I guess the takeaway is that the time you spend on planning is definitely not wasted, even though you are not getting anything done, you know, while you're planning. and I think it's very much a software engineering 101. You know, when you start your career as a software engineer, you just try to code, code, code, code and the more experience you get, the less you write code because you know that you should think a little bit more about it beforehand and, you know, plan a bit more carefully so you're more successful and I think the same lessons apply to vibe coding. So the more time you spend thinking about the problem you're working on, the more likely you are to succeed. You made a comment about even if the models stop getting better, you know, you can continue to significantly improve the products based on the current models. You know, that kind of raises a question for me about like how much of the, you know, performance gains in a system like Lovable like come from, you know, just directly model iteration versus like the things that you've, you know, built or done around those models in order to use them well. And so I guess maybe the starting place is like, how do you think about, you know, those things? Like, you know, some part of it is like prompt and context engineering. Some part of it is like the agentic loop. Perhaps some part of it, some part of that is like fine tuning. I'm curious how much of that you do. not to mention, you know, a bunch of infrastructure stuff. Like, you know, talk a little bit about, you know, how you take these generic models that, you know, are given to you by providers and, you know, allow them to fit into a system that's really focused on helping people build successfully. I think one thing that we have to admit is that the system you build must work in harmony with the models themselves. so you know a lot of the ai engineering which i guess is what i've been doing a lot for the past couple of years is really very much dictated by how the models behave and you know it's very much it's super empirical there you know i think even ml is largely empirical uh when you're working with different problems there's a lot of things that work where it's kind of unclear where it works and a lot of the ai engineering ideas i think are just you know working around limitations of models I think that's just what it is. And so we've been adapting constantly to newer generations of models and trying to harness the best out of these specific models. Like one thing I can mention is that in July, we released our first agent because until then, Lovable was not agentic. We were not agentic because agents didn't work before. That's pretty much the story. When I joined the company, the company was doing agents and it was doing a lot of agents we had a very advanced and super complex architecture with lots of agents and hierarchies and stuff and it just it didn't it didn't work like that's kind of what it was because the agents were not good enough at making decisions and you know all that stuff so we went back to something much simpler something that was more workflow-ish you know with some pre-processing steps preparing a bunch of things and then just trying to edit the code and then post-processing steps and etc etc and that worked a lot better and we the company took off not because of the specific change but things worked really well for us and we were having the most reliable results i think you could get at the time but then you just get new models coming out and they they actually work with agents and therefore you have to go back internally and say hey we need to throw away everything we've done And I think that's one of the largest challenges of working with LLMs these days is that you should not be attached to anything you've done. And you shouldn't have necessarily too strong beliefs that there is some sort of smarter architecture or anything. You just need to adapt to whatever you have. And now that the models are getting really good, I think is when things start getting pretty cool. Because a lot of the architectures that people were trying when GPT-4 came out, that just didn't work at all because it was not good enough. these architectures are starting to work and you can start thinking a little bit more about how can I harness intelligence and how would I structure anything assuming this sort of perfect intelligence that I can call, you know, anytime because the models are really getting there. And that I think is super cool. And so for us, what we're trying to do is to not be too attached to any architecture, just look at the current models we have. And what we've become pretty good at, I think, is just anticipate where things are going. And when we're building things internally, you know, it's like the big AAA video game studios. They are always building for the next generation of video game consoles, like the next PlayStation or something like that. They don't even have it, you know, but they just know it's coming and they usually do know specifications, I suppose. And they are just trying to prepare for it. And I guess we're kind of doing the same thing, except that we're in the dark. We don't know what's coming. but we know something better is coming soon. And you have to try to build something that doesn't work yet, but that's going to start working, which makes a lot of things very, very challenging. And it also makes you not fix some things that you know are going to be fixed naturally by the models. Because you have to think about your engineering investments. Like, would you rather fix something now that is just going to get fixed automatically? Or do you want to invest into the future? And there's always this trade-off where, you know, you need to make the product better today for your users. But at the same time, you should think a little bit bigger. And I think every agentic company has the same trade-offs to make. Can you speak a little bit more to the distinction between workflows and deterministic rules and all of that and kind of a true agentic loop, agentic system? yeah so so workflow and workflows and agents i think um yeah there's a lot of debates around these things which sometimes can be a bit frustrating but uh so to me like really agents are when you start letting models make explicit decisions around the next code path to take in some sense and workflows are when you have designed the algorithm you know yourself and you're just like sprinkling some intelligence in between i do think that there is a fine line between both. There are some scenarios where you have a workflow with different branches and therefore it's kind of an agent. You know, it doesn't really matter. But yeah, the agent has more this sort of looping scenario where you just go back to the same place over and over again. And I do think a lot of advanced systems out there are a bit of a mix of both where, you know, there's some agentic things and there are things that look a little bit more workflow-ish. And what I can say is that I think, like at Lovable, we don't think too much about this. We're just building a software system that tries to do the thing users ask for. But generally, the trend is to giving more power to the models, more power to the models, and just focusing on the feedback we give them anytime they take an action. You know, what is the most relevant thing we could give them now, given the action they just took? Can we give them more signal, you know, anything to help them go in the right direction? That's where I think there's still a lot of untapped potential. And thinking more concretely about a specific feature, for example, you know, I think one of the things that I that, you know, Lovable has done relatively well compared to other genetic coding tools is like a little bit, you know, more like front end friendly. Like, you know, you do an app in lovable and, you know, Cloud Code is probably going to be a little bit more aesthetically pleasing and lovable. You know, what goes into something like that? Is it like, you know, is it just context and a rag and giving it examples? Is it prompting? Is it, are there rules or some other like more structure that you're using to, you know, push the agent in that direction? So everything is context and some of it, you know, there's also a very blurry line between sometimes prompts and more like generic context. I mean, everything is part of the prompt in some sense in the instructions. And I think, I guess that's why we're talking about context engineering now, because it's just essentially all the stuff you put in there and how you can nudge the model towards something that looks better. speaking of I think the area of taste is extremely interesting for for front-end design and I think the labs are working super hard on this but it's a little bit harder to verify than some other you know topics but yeah what it means for us is trying to come up with you know how can how would you communicate first to human about design how would the designer think about these things how can you offload potentially the process of design to some more focused agents thinking about this you know trying to offload the context to something just more specific to it because I guess a good rule of thumb is that if you want to be top you know like like I guess for humans to not anthropomorphize too much but if you want to be really really good at something you probably need to focus all your time on that thing and I guess it's kind of true for for agents as well when you think about their time as their context. So there's a lot of room there for basically specialization and becoming, you know, as good as you can be. And then, yeah, obviously you can do a lot of retrieval, trying to bring relevant examples or anything like that. And, you know, you have your template as well. Your template is kind of a prompt in some sense, like the code you already have is teaching you how to write new code. So all these things matter in the end. And then it's not just someone writing a prompt that says, you know, do something beautiful. It's a lot more than that. You know, what I'm also hearing in there is that, like, simply it starts from making that something that the LLM, the agent, is thinking about or is considering. Like a user could, you know, put something very functional in the box, but the broader context that you're creating and giving to the LLM is telling it to, hey, also think about how this thing looks. And, you know, this is the aesthetic of things that generally Lovable is trying to produce and, you know, that kind of thing. It starts with pulling it into the focus. I mean, you know, we didn't talk about this, but obviously things like basic chain of thoughts are helping where what you just want to create is that you want the model to go into the right zone in some sense, you know, like try to bring some keywords that are related to what you want. just try to expand really, which I think is less necessary for thinking models because they are kind of programmed to do this a little bit. But you can still very much control the chain of other models to really bring them. Yeah, I think a lot about this as the right zone, like you just want to be surrounded by the right tokens, you know. If you think about the sort of latent space of all the things, all the thoughts you could have, you just want to be around all the things that speak about design, for example, you just want to be around that because you're more likely to go deeper. So there's a lot of work to make these things happen. And yeah, there's some work to think about what does it take to do something great about a specific topic? How would you even think about this for yourself? Again, like I anthropomorphize a lot, but you know, models are trained a lot on our inputs. So it does make sense that they need to go through the same thought process if there's any such thing. Yeah, I mean, this makes sense. I mean, it reminds me a lot of like when I see really good AI generated images, like those prompts are using words and ways to describe the image that never would have occurred to me. Like they're totally outside of my sphere. And, you know, just incorporating those terms, you know, in your words here kind of puts the model in the right design space for the output. Yeah, exactly. Yeah, talk a little bit about how to do this at scale. Like, you know, you were there at the beginning. Like, you lived through the hockey stick of, I think, what was the stat that in eight months, you guys got to 100 million ARR? Yeah. Yeah, so the... That sounds insane in terms of like what, you know, the duck paddling under the water. It is quite insane. And we have had so much scaling issues where there's a lot of hindsight always where you think, oh, we could have avoided this and that. But I mean, to be honest, I don't think we should have anticipated this growth because it would have meant we were not focusing on the product enough at some point. And scaling has been really hard on a lot of ways. There's a lot of scaling with, you know, understanding the user base also gets harder because we have all types of users nowadays using our apps where, you know, when I started, we were a lot more connected to the users we had and we understood them pretty well and the personas were slightly different. But on the engineering side, so we used to be a Python backend for Lovable when we started, and we've been pretty public about this, we're using Python because I think a lot of AI products were using Python because, you know, that's the language closest to where the ML libraries are. And I think at the time, we had no idea really what was going to happen. So Python felt like, oh, this is probably the right place. But we had a lot of struggles in production when things started to take off And I know we had reached our maximum capacity on our cloud provider We just could not get more instances and we were capping for many weeks We could not get more users while things were taking off. Like the, you know, the lines in our graph were kind of at 100% everywhere. Like we could not just get more loads. And we also, we happened to take down GitHub at some point in December maybe. and we had to migrate off GitHub in a few hours in the office without too much thinking. And I think that event itself gave us even more growth. So it was even kind of a curse. What do you mean you had too much load that you presented to GitHub API request limits or something? So at the time, we were backing every lovable project straight to GitHub. That was how it worked and everything was in GitHub. and we just simply started, we created hundreds of thousands of projects in GitHub and GitHub is not really meant for this sort of load where you have just so many projects under one organization essentially. So we kind of killed their database. And I think one of their on-call engineers just had to took us down at this point. And we didn't know anyone at GitHub and we are an EU company, so we don't have so many connections in the US. At the time, we didn't know too many people. So we really had to tweet, I think, hey, please, someone help us. And that tweet got very viral and that brought a lot of growth for us, ironically, in the next couple of weeks. And we've taken down a few providers. But yeah, the downside of this is you start having a lot more support requests. You have a lot of people coming to... And our product is also a product that, you know, it's not like, it's LLM based. So there's a lot of things that maybe do not work or not correct. And we, you know, we get so many requests of, oh, this thing is not working, this thing is not working. And we try to educate and that there can be users that are not happy, which is totally fair because they see the limits at some point. So that's been really, really tricky. And we had a couple of months where I don't think we did anything but just firefighting, trying to manage the scale. And now we are in, I would say, a better place. And the team has grown. All of this stuff happened where we were just a couple of engineers, so it was pretty tough. But now we're trying to hire some better engineers to fix everything, and we're in a more stable place. But one thing that's interesting as well, it's been really hard to get tokens from LLM providers because there's a proper shortage of GPU and it's real. It's not a lie. And, you know, we do inferencing at some pretty crazy scales. I think, I guess I can tell numbers, but we got some really like insane numbers when we had some events where the product was free for a weekend when the graphs just kept going up and up and up. and we've had to work a lot with the partners, you know, the labs to make sure we could get capacity. And that's been extremely hard to secure capacity as we've been growing so fast. And there's a lot of competition as well there, you know. And yeah, so that part has been extremely stressful as well. You know, if you cannot keep growing because you just cannot get the tokens, that kind of sucks. But I think, you know, everybody is working to make that easier for us. And now that we are kind of a larger company, we have better contacts at all these companies in the Silicon Valley that we couldn't have before. Maybe going back to this core user persona of someone who's trying to build something, do you think of them as being technical? Do you think of them as being not technical? Do you think of them as, today they're technical out of necessity, but you know in the future you want them to be non-technical like how do you think about that you know and part of that is like you know a lot of the things that we talked about that you have to be thinking about or know how to do or be second nature like are fairly technical things and uh if you're uh if you're you know not technical like it's very easy to get started but it's also very easy to end up in a blind alley or in a hole and like you know it seems like that will cause a lot of frustration. Yeah, I think it's funny. So most of our users are non-technical and or they start being non-technical. But, you know, when you start programming, like just to keep on with my analogy from before, when you start programming, you are non-technical by definition because you don't know anything yet. You know, everybody starts at some point and they are non-technical. And you can learn to be good at lovable or vibe coding the same way you can learn to be good at programming. And I think it's a bit underestimated that, you know, people learn. It's not because they're non-technical that they don't know or they, you know, like being non-technical is not a bad thing. It's not a fixed state, right? Yes. And it's also not a bad thing. You can be brilliant, but non-technical, like these things are totally orthogonal, right? So a lot of our users, they start and they don't really know how to reason about building things because I do think that you're right. This is a skill. It's a skill that you learn when you do software engineering or product management or project management, I guess, in your professional career, right? And they learn to do these things. And there are a lot of resources online on, you know, like lovable courses and how to think about things. And so the same way you learn to program, you can learn how to build things, how to do product management. So it's not really true that you're not doomed at all if you're non-technical. you can really learn to be lovable, technical, you know, whatever you call this, really. You can learn the skill of understanding planning and also being a builder in general. You know, like if you're trying to get users, how to go out there and get users and decide when it's time. You know, what's MVP, all that stuff. That's true. And, you know, I'm willing to concede that. But that's very different from a lot of what I run into is like, yeah, there are five copies of a function that all do kind of the same thing. And like, this is going to be a nightmare. Or like, why is state handled in this way? That's going to be a nightmare. Or like, why is the function have these arguments? Like, that's kind of dumb. Like, those are like, those aren't project management things. Like those are software engineering-y kinds of concerns. And I just, you know, I think there, you know, I can anticipate that one of the answers is the models are just going to get better and no one's going to have to think about this. But that also, to me, like, you know, it concedes a lot, I think. I can speak for this because I mean, you're right. I never want to pretend things are just working and it's magical and we've solved AGI, right? Yeah, yeah. But one thing that's true is that a lot of our users learn to code also with Lovable. We have some power users from the old days before when the product was called GPT Engineer that have been using it for a while and they were totally non-technical and over time, you know, they've learned to look into the code and try to understand what was going on and they've really learned by looking at code being written and then obviously then you need to have the high intent, you know, to learn something. But they, you know, you can ask lovable. I mean, that's one of the cool things. You can interact with this code and ask questions about it and they have learned to look more and more and then they have become these hybrids, I don't know, developers, vibe coders, whatever you call them, but they are technical and they probably could not write code without any AI help. They would probably have, they would not even know how to start. But with the help of AI, they know how to guide and they've learned how to architect things to some extent. And that's really cool. And some of them become full-fledged programmers. You know, they just love it and they fall into this. And like, you could be cynical and say that it's just saying that, yeah, vibe coding doesn't work, but I don't think it's true. I think first, it's amazing that people find their way through coding, learn to code through this angle. I think it's super cool. I wish I could have learned to code like this, right? It's like someone always willing to help you. But I mean, I want to interrupt you because that's a really interesting point. I think when you're learning to code and you're not coming at it from an academic perspective or like in a program where, you know, you have this external pressure to learn to code, the advice I think that's most successful, you know, besides from like do it with other people is, you know, do it in the context of a problem that you care about and that you find interesting. And if you don't know anything pre-vibe coding, like there was nowhere to start. Like you could, you know, find a GitHub that was kind of similar to what you were doing, but chances are it was like way too complex and, you know, beyond there's like, you know, I think vibe coding does enable this kind of lift yourself up from your bootstraps in a sense of like you can, you know, create out of nothing this, you know, this thing that you're interested in and it gets, you know, far enough along that you can start to inquire about it and, you know, reason about the way that the agent has like decided to build it and then you know maybe try to decide different things at some point or you know learn some terminology and explanations and then research that technology and you know then that you know becomes the you know the the ai the gen ai like you know rabbit holes that we all get into and we're learning things about you know things that we don't know a lot about. We have lots of users who are, it's like a new breed of programmers where they know a lot of technical terms, they understand the concepts, but they actually don't understand a lot of the low-level stuff. And it can be a little bit jarring to communicate with such users because they have a very different model on how things work. They just learned, and it's like they just learned enough to do the things they wanted to do. So they understood what abstractions mattered and what things didn't matter. And I mean, it's absolutely true that a lot of the code still written by LLMs is not good enough, especially at larger scales or where things get more complex. You know, it's like harder to build upon, you know, like you add layers and layers of complexity, right? And I don't think that's only on LLMs at all. I think the agent frameworks that we have needs to mitigate against this and to find ways to enforce some sort of rules and have the right guardrails. That's for sure. I really don't want to pretend that everything just works because if everything just worked, then everything would be over in some sense. So there's still a lot of work. There's still a lot of work, but it's getting there. Yeah, one more topic that I want to take on with you, and it's along the lines of this, or it's related to this technical versus non-technical user is the idea of evals as being a core element of success in building with Gen.AI systems. There are takes on either side of that. We were talking a little bit about that earlier. To what extent do you think that evals and metrics are something that is important for your users to understand how to use? Like certainly, you've got evals that you're using internally in the building of the service, but in using an authentic coding tool, do you have to be thinking about evals? Maybe it depends on what you're building. Talk a little bit about that from the user perspective. Yeah, so I think no matter what you build when you are building anything, any kind of software, you just have to wonder how am I going to know that this is working? You know, like, actually, I think even that in itself is a bit of a, again, going back to product management skill or software engineering skill, like learning to ask yourself, wait, why would I be doing this? And then once you've answered this question, like, how would I know that I did the thing I wanted to do? And I think for a lot of problems out there, that's when you should stop because you realize that, no, actually, I don't know if it's worth doing or, you know, how am I going to know if it worked? So, you know, in software engineering, it's called tests, right? You can write tests. You can test your software through multiple ways. You can click around. That's the way of testing, I suppose. Or you can write proper tests that will be, you know, run over time. And this is how you verify that things do not regress, which is super important. and when you're doing anything with agents or building any AI-based application, it's the same thing. Like, how do you know that it works? And then the texts I hear online are, well, you'd know that it works because you just tried the thing, you know, you build it and you vibe with it. And I think I support that, sure. But how would you know that it stopped working? How would you know if things break, right? When we were building Lovable, when I joined a year ago, we didn't have any evas or anything. It was just vibes everywhere, you know? And that's great. Like, it's great when you start and you're trying to get the feeling of your product or your UX, whatever you're building in place. But once you have that in place, you want to pin the performance. You want to lock the performance, right? And I think that's where evas come in and you just want to say, I want to make sure that things do not stop working. And if they break, I want to know about it. The same way you write tests, You know, when you do software engineering, you have continuous integration running the tests and telling you, oh, this one is not working since you made that change. You want the same thing with GNI applications. So that's one thing. And then the other thing, when you reach a scale like ours or you have a product that's a bit more general, is that our users are doing every possible thing you could ever imagine. You know, they are building projects connected to any API, any service, using any library, anything like that. so we want to start expanding the coverage of our evals to be you know looking at okay i'm going to look at this library i'm going to look at this use case that this you know five percent of users are building this thing i want to make sure that they have a good experience etc etc and once you have great evals what's pretty cool is that you can change your i don't know your prompts or your models or whatever and get confidence that things are actually going to work better and visit your users. That's one level. And the ultimate thing that everybody wants to do, I think, is that you want to build a self-improving product. You know, the same way the AGL labs are always talking about, you know, building this sort of self-improvement loops where the next generation gets better. Yeah. And we want to do this as well. We have more users using the product. We get more data. We can analyze the data. We can understand what we should be changing. We can extract data sets for doing evals, we do the evals, and so on and so forth. And you have this sort of thing that evolves over time and runs optimizations. And I think, you know, everybody wants to do this. In practice, there's a lot of things that need to go right for this to be fully automatic. But if you don't have that, and I do think in the vibe coding space, actually, next year is the year where we're going to see who has been doing the evals seriously. I think basically your products are going to fall apart if you don't have that. Because as you get more more complexity. And are you talking about the vibe coding company's products or the products that the users are creating with those products? I mean, I think both to some extent, you know, because I think the incentives are aligned there. So I think both. And as these products become more and more powerful, like, you know, we're going to be able to do literally anything in Lovable at some point. How will you be able to, you know, there's no way to vibe check this product anymore because it does all the things like, are you going to try everything? You absolutely cannot do that. So you need to have some sort of guarantees that you're not breaking things for some users. And evals are really hard. Like I think the labs have not figured them out, you know, and it's just, it's still a small subset of what people are doing no matter what you do. Awesome. Awesome. Well, Alex, this has been a super interesting chat. It's been great catching up with you and talking a little bit about what Lovable's been up to and what you're seeing out there. Yeah, thank you so much for having me. It was super fun. Thanks so much. Thank you.