SkyDiscover：AIがコードだけでなくシステム全体を作る時代へ（UC Berkeley Sky Computing Lab）

Alex, Mert, Shu, Shovem, welcome to the Weekday Podcast. Thank you so much for joining. Thank you so much for joining us. Thanks for hosting us. Awesome. Well, we've got a packed house. I think the work behind Sky Discover is just super captivating. I'm so excited to dive into the story. Maybe especially because we have sort of a big panel here. Do you mind each maybe giving a little bit of your background and how you came to be working on Sky Discover? Okay, maybe I can get started. Hello, my name is Xu. I'm currently a fifth-year PhD student at Sky Computing Lab, advised by Yon Stoika. And then my work has been focusing on previously mostly about like distributed systems, storage systems, and recently moving towards more of like RIM systems and like how AI can solve this like complicated, more and more complicated like systems task. Yeah. And I, maybe I'll go next. I'm also a fifth-year. I'm Alex Krensel. I'm a fifth-year advisor by Sylvia Ratnasamy. I work pretty closely with Scott Schenker and Jan Stoica.

I kind of come from core networking and core network systems, global-scale internet control systems and internet architecture. But especially the last year, I've shifted into AI-driven discovery and now with the Sky Discovery team into AI-driven synthesis. Okay. Hi. I'm Matt. I'm a fourth-year PhD student at UC Berkeley EECS. I'm co-advised by Ian Soika, Alex Demakis, and Kanan Namchandran. My work is on ML systems, and particularly like these test time scaling algorithms, how can we make them faster, more efficient, but also more performant by ways of either making the exciting, speculative decoding-like methods or making them more performant with these evolutionary algorithms. and like we have at Sky Discover. Yeah, pretty excited to be here. Thanks for hosting us. Hey, I'm Shubham. I'm a second year PhD student, just turned second year.

And yeah, I've been working on machine learning systems as well and recently more focused on using agents for building systems and also in the area of using like formal techniques and things like that to actually guarantee that the systems built are practically used. Pretty excited to be here. Yeah, amazing. I mean, I've started seeing a bit about this evolution into AI building systems coming out of Berkeley. I think like probably the cat's out of the bag. Everyone knows AI can code really well. We've had SWE bench and all these different coding tasks. But this evolution into building entire systems, I think it's just such an amazing concept. Do you mind maybe now like setting the stage on maybe just what does it mean for AI to build systems? What would you like to? Maybe Alex can Yeah, that would be like I think we've seen this transition where the very earliest AI systems if you think back to Copilot were like just trying to auto-complete your lines and even that was pretty magical, you would start typing and figure out okay, here's the remainder of the variable

you're trying to assign or whatever and then we got a little bit further along and you'd have a comment and it would synthesize a function and then I think the last year or two, especially maybe starting with Cloud Code was last February, was it? A year and a half ago. We started to see what could happen if you exposed files and a file level view of coding. But I think even then, people still were going to these coding systems and saying, hey, help me build out a particular function or particular piece of functionality. I think the Sky Discover team, we were really looking at this. As you can tell, we all are really coming from a system's backgrounds. And you get this difference where there's a big difference between a piece of code, an algorithm, and a system. In a system, you start to kind of reason about how you manage complexity by putting it into kind of layers of abstraction. And you want to do this because systems are not kind of a static thing. You design a system to be

scalable, to be extensible, to be interpretable, to be kind of able to run in production. And it's kind of an order of more complexity kind of reasoning about a task like that. But we were looking at where these coding systems are headed and we're kind of thinking and following that line and watching it go from line completion to functions of file level. And so we were really inspired by the thought of these things that today take kind of PhD level expertise to go build a database, a inference engine, a KV store. If we were to really like trace out where these things are headed, what would it look like to go and optimize existing systems of that kind of complexity or synthesize systems to that scale. Yeah, that's a little bit of how we're thinking about it. Yeah. And I would just like to add on, like, for example, since last year, like at the very beginning of when these models are coming out, we're seeing these models solving more and more complicated, let's say, like math tasks and very simple coding tasks.

And now, like, as people are starting to use all of these coding agents more and more often, and these coding agents are powerful in generating, like, thousands of lines of code, we're thinking about, okay, like, what about daily software? Like, let's say, the databases that we use, the storage systems that we build, or even, like, the inference stack, the post-training stack that people are particularly optimizing for their own requirements and workloads. How can AI automate that and optimizing that entire stack according to my requirements? And how can we trust, like, whatever AI is generated? Like, you can always see, like, those agents are generating a bunch of tests and say, like, okay, agents passed, like, 300 out of 300 tests. But it's really, like, trustworthy. So when AI are, like, expanding, you can use, like, multi-agents and everything to scale up the amount of code that you generate. How do you actually, like, make sure that it's actually performance and actually work as you expect it? Yeah. So that's why we shifted from a little bit. So when we started Sky Discover, we started about optimizing algorithms, optimizing, let's say, scheduling algorithms, congestion control, et cetera,

some of the systems problems that we have, and moving more towards end-to-end software stack and look at what are the challenges up there. Yeah, I mean, one comment to add there is like, in fact, in our work, one focus that we had seen is this thing called just-in-time system. So imagine that you now are basically capable of building whatever you used to use from some open source repo. But of course, the open source repo comes with something we call generality tags, which is basically that you are using something that has been built for hundreds of use cases. But maybe you know your use case well, you know your workload well. In fact, like in a couple of days, we might be able to show you something more where we are thinking that the inference engines that we built for on-device things. In fact, MERT is leading one of them where we are seeing that if you make your inference engine for a very specific Mac, Metal machine and for a model and a workload, you can do a lot of optimizations that you would not be able to do because you were doing things for a general setup.

And that gives you that unlocking power. But because you're using agents now, it's very, very brittle. There are like hundreds of hacks agents do. And there are a lot of things that agents will just find ways to break things and exploit the gaps that we see on the requirement side and on the model side, which is like the world where the final thing is going to deploy. So given those gaps, the SkySynth project is some sort of our angle to tackle that where we are trying to make sure that the agents do the right work. and don't try to exploit the gaps which are already there in any software system. Yeah, so I guess if I can maybe start prying, there's so many interesting ideas in this, but if I can start prying maybe from a sort of selfish angle, sort of from the perspective of WeV8 and our vector databases, we make a lot of this argument of, you know, we have a specialized vector database. It's a database that's customized to, you know, vector search and vector storage from how you implement a custom LSM to the custom indexes.

And so I guess we had a podcast with Andy Pavlo from Carnegie Mellon, who's a well-known database professor, who was explaining this vision of self-driving databases. And I always imagine it sort of as hyperparameters, say like with the HNSW vector index, you have like the EF parameter, like these little index tuning parameters. Or you can have like how often the LSM is going to compact, or, you know, how your replication is configured. So I guess I'm just curious about if we can maybe start on the database example. I know the paper describes, like, a customized key value store. Maybe just I'd love to understand how SkySynth could create a database for a specialized workload and what you think the future is of tailor-made database systems. So I think that's a very good question that I have. So we started working on KeyValue Store, mainly because KeyValue Store has many, many diverse different kinds of properties. So you can think about it as like, okay, like how many requests do you have? And in this request, like how many are read requests and how many are write requests?

And then among these requirements of like building a KeyValue Store, like what is your amount of resources that is available? Are you building like a single machine key value store? Or are you building it in a distributed setup that you need to manage the consistency and stuff? And also, what is your consistency requirements? When you actually get back the value for your key, can you tolerate some staleness so that you can maybe have better latency to answer your request? So there are many different dimensions of properties here. So this can be separated into, let's say, those requirements. So let's say, how reliable is the system? How reliable is your key value store or databases? What if it fell? How long would it take to actually recover? So this is one perspective. And second perspective, as I talked about before, is consistency. Like what kind of consistency level or requirements do you require for your setup? And also, what is your objective? Like are you going to optimize for throughput or are you going to optimize for latency? And what is your workload scenarios? So this has like very diverse range or diverse dimensions of like all of these requirements and things that we can potentially specialize.

So for example, if all of your requests are like hotkeys, then you can design your recache in a way that it's like caching all of the hotkeys in memory so that you don't need to actually fetch it from, let's say, from disk or something like that. But if all of your requests are sort of like a write request, then you design something completely different compared to like a rehab request. So we started exploring basically taking existing traces. So some of the existing traces are just like basic traces, but tuning in different ways that it has different rate and rate ratio. And then also we have some production traces we take from Tencent and also all of the other Twitter traces as well. And then we see whether, if given that you know this workload information and you know what is the objective functions you're going to optimize for, and can AI actually generate something that is better, let's say, comparing to RocksDB or Redis and different stuff. And actually, indeed, like those agents are coming out with new designs,

different sort of caching designs in different sort of workloads and requirements. And also like different sort of, let's say, hierarchical memory management. You put different sort of like size of the cache in memory or different size of the things that you put on disk. And then also for consistency management, then like how do you make sure that it's actually correct to maintain, let's say, way up to right consistency or strong consistency if you expected that. So we curated, I think, around six to eight different kinds of traces. And then we see, if given this information for this particularly specialized workloads, whether the agents can specialize for that. And actually it's achieved up to four times better performance comparing to those even very specialized systems from, let's say, Microsoft. And yeah, see good performance there. I guess my first question is, well, I guess I don't want to, like, I really want to kind of talk about, like, I'm worried this pivots the topic out too much, but I guess, like, would it build onto an existing system?

Like, do you think like the specific, maybe let me just ask, like, so I think about like kind of this as being like a DSP for systems or, you know, specifications, I think is a paper that came from Berkeley and Jared Davis and this idea of, yeah, like abstracting how you design a system. So you like I guess it almost makes me ask like what is software now now What happens to the artifact of the system that created Yeah, we're really curious about this question, too. What is the artifact now? Look, if you take our vision to the extreme, you're hitting on something really interesting here. If you take our vision to the extreme, you get to a point where the artifact is kind of a unique function. It's a function of the traces that you provide. and the reason that we've had our Outerfacts before, why these things have stood on their own, why you have something like Redis, is because it used to take years and years to build these systems. It takes years to train someone to understand. It takes a team of engineers or a team of PhD students

five years to go build a system, harden it, get it to the point where you can trust it in production and then no one is going to go pick it and build a brand new system unless you're meta or Google and you have this huge scale and the savings you can get are actually worth the millions of dollars that it costs you to develop your custom system. Otherwise, it doesn't make sense. It's not sane sense for anyone else to do. I think we're trying to build a system of saying, if you could, if you could hire a bunch of people to build a custom database just for you, for your exact kind of things that you want to store, you can get better performance, but it used to cost you so much. And we're driving that cost way down with coding agents and with the frameworks that we have. And I think, as Chu very correctly put earlier, the remaining problem is, can you trust the system that we've built? And a lot of our research the last four months has really been on, how do we prove to you that the system that we've built is correct? either by tests, an extensive test suite that's evolving, or actually by formal specification, formal verification, like a guarantee that this is correct. Yes, I think that's maybe the topic that Ben dives deeper into details

is how you think about testing. And I'm sorry if we stay in the database. If you don't mind, if we kind of frame this mostly through databases, that's all that I understand. I mean, maybe later we could talk about how this might differ for a task queue versus a database. but if we can stay on the database thing, and I guess maybe let me backtrack and ask, how do you think about the specification? How does someone come in and tell you about what kind of workload they have, what kind of database they need? So I think there are some requirements that are about environments that every user or whatever they are trying to build something for their environments, they have the environments in mind. So let's say how many machines do you have, and each of the machine configuration, like what is the memory size and what is the disk size. disk size that you can put. So this is definitely something that user will provide in default. And if you think about different domains, actually different domains have different properties that a user might want to specify. So it can go from very high levels. So let's say I just talk about reliability. For example, if your database or if your data systems fail, how long do you expect it

to recover? And what is the data loss that you can tolerate in this case? Maybe most of the time Like people won't tolerate like data loss for any sort of like databases that you create. And so, for example, reliability. And second thing is about consistency. Like when users, they have their requests, like when they read their stuff, like what is the staleness of the data that it's being read? Like if you can tolerate more about staleness, maybe I can give you a better performance from the database that I create, right? So this is another dimension. And about the workloads itself, like sometimes people, when they build their systems, they have their workloads in mind. So let's say, particularly let's say for the key value story in our cases, maybe you know what is the rough rate-wide ratio. What is the recency or frequency of a particular key in your workloads that you can optimize your assistance for, right? So as I talk about, like, if you have a very, very heavy workload, it will be completely different than optimizing for a very, very heavy workload for a different consistency level. So I think there are some basic characteristics about, let's say, the workloads that you have, the requirements that you have, and also maybe including the objective functions.

Maybe you know that you want to just minimize the user latency SLOs and you have some particular constraint over, like, I'm not going to violate this SLO while building my system. So there are basic properties that, let's say, it's probably domain knowledge or expert knowledge in a particular domain. So let's say databases, I just list down a couple of things. And maybe in storage systems, it will be something different. And in inference systems, it will be completely different, right? So probably, like, in that case, it's like there's consistency is like something in a different notion. And correctness is a different notion as well. So how we learn from this, like, what is considered as, what I should provide in specification is very domain-specific. And probably people have built, like, years of systems that follow this routine, right? So let's say when you write a database paper about, like, building a database engine, Maybe you write about this particular property about fault tolerance, about reliability, about consistency, about performance, metrics. So from either the codes or the literature, you know sort of like what are the basic properties that you want to post.

And if the user doesn't know, I should probably talk about, okay, I should specify my consistency level. Then the AI should be able to post the question and say like, oh, so what is your requirements? Like, what is your constraint here? Like, it shouldn't like assume something that the user doesn't even say. So I feel like the gap between, let's say, if you just naively use CloudCo, it might go ahead and do something that you don't expect. But then, of course, if you have a good pipeline and you know the domain knowledge and what people have been built before, then you know that there are several principles and properties about the domain of this artifact that you should ask and you should clarify before proceeding to build a brand new system. So I think this is the spec that we're trying to build, and we're trying to build the spec in a way that is like follow domain knowledge, And when we shouldn't assume something, you should pose the question and just proceed building like a specialized systems from there. And I think this can be, let's say, I'm talking about a way that, let's say, I'm just daily user of like hot code.

I'm typing like, oh, I want to build a database for my word code. And I'm expecting hot code to ask like, oh, so what do you want to optimize for? What is your consistency requirements? What is your reliability requirements, et cetera? But, of course, like, Siobhan can talk more about how do you actually, like, if you are a domain expert already and you know what you want to write, you probably can write, like, a formal SPAC. And then, let's say, like, the coding agents, according to the formal SPAC, can generate some sort of verification or, like, formal proof so that it proves that it completely satisfies the SPAC. Yeah. Yeah, completely agree with this. I think one more point to highlight here is that we, again, given that we were using this tool to generate systems, we kind of understood some pain points about that. Okay, writing spec is very difficult. Like you can't expect a user to come and tell the entire routine or entire laundry list of everything that they might expect from a system. So in the pipeline, again, if you go for clawed code, it might, and if you prompt it well, it might actually ask you questions or it might show that it can ask questions.

So it might like show of that capability. But one thing that we realized was that when an engineer actually builds a system, So when the code development happen, then there are a lot of gaps that start surfacing. And when those gaps start surfacing, Claw always takes the cheaper way. For example, if you ask it to optimize for performance, it will take that assumption where performance optimization was easy to do and it will not care if it is hacking. Very simple example, we started using some YCSB traces and things like that. And again, if you don't know about the trace, that trace is more or less deterministic, where given the key, you can actually predict or not even predict, you can map the value from the key. The initial systems that we built, it was like 30 times throughput, 40 times throughput. And of course, without seeing anything, we were like, okay, I think we have built something. But as we got into the details, it was very clear that Claude was basically hacking it. And it was a very obvious, you can imagine, it's like an axiom that in a system, I am not going to give you keys where the values can be determined. It's all about storing those values. and then our insight there was that when the code development happens,

then only there are more and more holes that are going to be exposed and our synthesis system, which is like SkySynth, is able to surface the right level of abstraction that the user should be able to answer. So, again, we do two types of generation. One is the generation where you are actually starting with requirements, as she was mentioning, where the system designer knows the requirements that they want at some level, but they might not be the people who can actually do things in a formal way. By formal way, you can imagine if I ask you to write a sorting function, of course, you will be having some properties that, okay, my negative number should be on the left-hand side, my positive number should be on the right-hand side. If there are duplicates, then this is how I'm going to handle. So you can give some high-level specifications to what you want. But if you are doing things in a very critical domain, for example, in databases, of course, you know, read or write, or there are like properties that you care about those properties there are sometimes those properties are actually sometimes very hard to define in natural language because it is not

clear because you are talking about like concurrency distribution and it's not easy unless you start defining things in a slightly mathematical format that we call formal language like that is what lean and rock used to do so in our pipeline the thing is that when a user comes with a system design or comes with something which they want to build the the agentic system is going to probe the user with the right level of questions and take the route of either the formal way or the test driven way and in that then again our loop is basically designed to keep on finding out what the gaps the user missed and as in when those assumptions need the user the user will be probed otherwise right level of assumptions will be made so that the system can be built more robustly. So the optimization pressure is something that leads to reward hacking. And SkySynth is somewhat a vaccine that basically prevents the system from doing a lot of reward. Yeah, it kind of reminds me of like Cloud Code. Like, you know, it writes some tests.

It's like I've passed the test that I also just wrote and that kind of thing. And then, I mean, it's just amazing thinking about like it still sounds like you have the sort of like classic deep learning, machine learning, like training data, testing data. But now the artifact instead of a trained model is code, a system. And then the data is also like more persistent, I think, than you, well, in the sort of search and information retrieval sense, you kind of have like this persistence of like the corpus and that maybe comes into this territory a bit. But I guess like kind of maybe talking about testing and you mentioned like formal proof, I guess it just makes me think like you know if I say I need you to sort these numbers and okay so you know it's going to either maybe bubble sort it or it's going to quick sort it it's like how do you then like how do I know the difference between if this system has done either bubble sort or quick sort yeah so these type of things ideally so you should specify when you first ask like skyscraper

synthesize these things. But our system first tries to understand what you implied in your task specification. So if you asked it to come up with a sorting algorithm, probably it would first ask you back, okay, do you have any preferences on the time complexity of this? then if you say nothing okay just give me a quick one it will do some literature review and try to see what other code bases exist in that domain then it will try to implement what it thinks is the best given your specifications you provided so basically like the task should be tied with whatever you specify in your properties right so for example like let's say if you don't formally prove your, let's say, consistency or whatever other thought tolerance level, you probably want to stress test a little bit

about how thought tolerant is the system, right? You can send a bunch of requests, like many, many concurrent requests as well, and see whether it will just fail. And see after when it fails, how long does it take to recover as well? So I think people have this task-driven development things in production system as well, because they cannot formally specify each part with each component of the end-to-end system. And this stress testing is tied with particular properties that you desire. And you typically want to see, like, okay, how long does it take to recover, whether it does, like, garbage collection, and when workflow shifts it, it's, like, completely, like, breaking down or something like that. So I think, yeah, so the test is, like, basically tied with whatever you specify and whatever the users want in their specification. Yeah. And in general again you put that point very correct that does cloud will basically tell you that I generated 100 tests and the 100 tests are passing because writing good quality tests that are actually capturing those specifications is the key

And in the pipeline, again, the idea is that every test that is added is actually enforcing that spec. So again, there are some details here that the test will have to prove that it is actually testing the particular specification by generating, say, a reference implementation, which is correct on that aspect and incorrect on that aspect. And if it is actually able to differentiate this, then only that test is actually like a gate that can allow a system to pass or not pass. So it's more like a rigorous way to do things because the domain actually requires you to take care about those specifications. Yeah. I mean, I guess like I'm really interested in, it just inspires me thinking about like Python versus like Go and just how you think about the abstraction of what the system is working with because if it's say something like go or maybe i know rust is maybe more popular but we use go we get so that's what i mean by like slightly lower language but um like and it's looking at something like heap allocation and just i guess that abstraction is quite different like when you're i don't know if you want to call

this vibe coding or like what we call sort of lm coding but like what language it's using on top of that really, I think, influences just, like, how it works, these specifications. So I guess I'm just curious how you think about maybe even just, like, programming languages with this. Yeah, I mean, again, good question. So one thing to take care of this when you're building a system is, of course, you can either have preferences, that is one thing, but if you don't have a preference, then the goal of the system is that it is optimizing for a particular workload. So you really care about performance in that part. And when you are caring about performance, when it starts getting into like the blind spots there, there would be some languages that might be easy to build in terms of closing those blind spots. For example, if you are caring about something where memory safety is actually a thing, the system will steer the agent to use that memory safe language like Rust because the system is going to be built more carefully

and it will be, it will have a hard time compiling and hence whatever is built is going to be memory safe. But maybe the performance will get compromised. So in fact, the language is also something that is very attached to the performance that you need, the specifications that you care about and to make those specifications abide by you are basically writing tests. So, so the whole thing is that make for example, if you do kernel engineering, like a very simple example, it is very easy to write a correct kernel when you start with a high level say PyTorch or Pythonish language. But then all you care about is performance. Now, if you are dealing with a TPU versus say X version GPU, you want to go into that stack where it allows you to write the things where you can say fuse the kernels very effectively. Or if it's like a quantized kernel, you are caring about how the precision errors can be minimized. So it's really, really an implementation detail. The users should not be caring about it unless they have a preference. And that's how our system currently actually helps the user. that unless and until this is an assumption that is leading to a path where the specification needs to be changed here or there,

the user will not be even exposed with that. He will be given that implementation that is ideally most performant and least reward hacking. By least reward hacking, if it's a test route, it's more about where the tests can be done very, very properly. As I said, Rush will be more, more favorable if it's like a memory safe thing. But if there are things where you don't care about it and you can get better performance, not caring about your compile time memory safety requirements, you will go in a C route or something. And on the other side, in the formal space, again, there are like Lean and Rock, which acts in the model world where you are trying to prove things. In the downstream, again, you either try doing some old style OCaml kind of languages. but in the latest world, again, you have CompCert kind of compilers, which is in the C world, plus Verus kind of things, which is in the Rust world. So again, these are like very, very intricately attached to what you really care about because you can write everything in every language. What you choose will basically decide how easy it was to test it and how performant it was at the end.

So it's really depending on your objective that you care about. Yeah, you're making me want to brush up on my understanding of compilers. It's been a while since. And I guess, like, yeah, so I guess my question is, like, you know, from the specifications, you could just go all the way into maybe generating, like, a binary, I guess, like, in a super crazy way. Like, do you think that's ever – there's always had to be some intermediate code artifact? Again, very good question. So the whole thing here is that if you give me a specification and you care all about performance, again, at the end of the day, you really, really want a binary that was not built off any compiler that was making any assumptions to speed up your implementation in a general way. So you really want your agent to actually write you like assembly code. If not binary, that is like very, very performant for your system. And again, if it knows your workload,

it might be doing some prefetching, some things that can't be done on the higher level languages. You want to go there, but obviously like there is a cost aspect attached there also because you would require the LLM to iterate more and more and more to get to that level. So it's somewhere where you basically close your boundary saying that, okay, I'm fine running this experiment or running this system building task for extra hours and I'll spend n number of dollars on this. In that time, you really want your system to be there. But again, once you have a version, LLMs can be made to work without eating and sleeping. So in an ideal world, you can just make them optimize the thing more. And in the optimization, again, when the language becomes a barrier that, okay, now to optimize further, I just need to go to a lower level language. It's, again, like kernel engineering. Once the idea of how to write flash attention gets invented, you really need to go into the lower level language to actually do the memory fetching because Python or PyTorps won't allow you to do that. But, yeah, I mean, I hope I answered you the question, but that's the point.

Yeah, I mean, I hope I'm keeping the conversation on track. But yeah, it just makes me think about like right now, I think you, I mean, you have to be pretty cracked as they say to like be managing a repo and like multiple like language abstractions. I think that's kind of where this is taking me. Like if I had a Python project that also has like a, like a parts, I mean, yeah, you have like connection. Sorry, it's been a while since I've done projects like this where you would like connect like C into Go. So it's just been a while since I've really thought about that kind of thing. But, yeah, cool. And I guess I'm still just kind of thinking about, like, you know, like building on top of, say, Postgres or, you know, WeV8's open source, like just how you think about extending open source, say, databases and just what role that has to play in this research. so maybe like I think we're still actively

thinking about like in what cases does it make sense to synthesize something completely from scratch versus like optimizing some existing systems so of course like synthesizing something completely from scratch can take probably a lot of time so there may be like very mature production ready systems maybe they're like they have They are built for general workloads. They have many, many tasks building in their original systems that is open source or closed source. And so it takes less time to build those components. But probably synthesizing from scratch can allow you to jump out of the local optimum that is being heavily constrained by whatever structure that the general systems has. So let's say if I were to build on a general system, maybe I first need to understand 40K lines of code for the general system to be able to see like, oh, maybe tuning this parameter will work for this kind of specialized work codes. Maybe tuning a combination of those two configurations will work for my particular requirements. So, of course, like it takes time for understanding the code basis

and probably the structure or the abstraction that the general systems has might not be the best for your specialized use cases. And it's unclear that like whether one case optimizing it, it actually can achieve better performance compared to synthesizing something completely from scratch. And we actually try, like, in our key value store use cases of, like, building on top of, like, Microsoft faster and see whether it can, like, do it well. But then comparing to our synthesizing from scratch experiments, it's actually achieved less throughputs comparing to just, like, building something completely from scratch. And, of course, like a time constraint. Like, sometimes coding agents can get stuck in a local optimate and thinking about, oh, this abstraction is everything that I have, so I'm just going to tune partial or components of the system. But versus when it thinks something completely fresh, maybe it can come up with a better design from thinking about backing up in a high level, like how should I design my memory, how should I coordinate between memory and disk,

and how should I handle all of the requirements that a user have. So ideally, I think the system that we're trying to build is knowing when to synthesize, knowing what to optimize, but then at the same time ensuring that everything, Either you do optimization or synthesize from scratch, you want to make sure that you don't break some of the guarantees that the systems should have, what the user specifies in their specification. So correctness still matters even if you are just optimizing for a particular system. So let's say if you change the caching algorithm or the caching components of Redis, and then actually something breaks whenever you input a new word code. It's not correct. So I think in both cases, we should think about, like, okay, so what are the right set of tasks and what are the right set of guarantees that we should provide to the user so that the user would trust the ultimate artifact that are being generated by the agents? And second thing is, of course, performance. Like, given, like, a limited, let's say, time budgets and also cost constraints, is it better that I start from, let's say, existing well-known system?

If so, start from what system? Because, like, many different, there are many sort of, like, open source, like, databases out there. and which one should I get started with versus completely synthesizing from scratch. I think those questions, as models become more powerful, hopefully the model will know what kind of high-level spec that each system will provide and whether each system will have specialized strength in terms of optimizing for a particular workload so that it can dynamically choose between, okay, I am going to optimize the system and I'm going to choose this system to get started with because it has this kind of properties that it's easy for me to optimize. Or I'm just going to completely start something from scratch where I see, let's say, some features are completely not supported in any of the existing open source databases. So I might as well just build my own design. I have a whiteboard. Maybe I can draw my key design idea to optimize for this workload. And I'm just going to synthesize something completely from scratch. So hopefully this will be more automated.

and I think, yeah, so we're trying to build something that can achieve this, but the goal here is being both correct and also being very specialized in performance. I want to call out here just how surprising this take is. I think it was surprising for us when we got to it too, but I think for the current stage, just aware of the field of that, if you're working in AI, you shouldn't be designing or building or thinking of what you're doing as for today. You should be thinking if you look and follow the trends of where they're headed, you have to skate to where the puck is going, not where it is. And so there's this really weird thing, right, where it's like it's counterintuitive to think that you should start from scratch to build your system if there's an existing system out there. But if you assume that the model is getting better and better at building and reasoning and designing, it's so easy for it to even get to any particular design. You should just let it start from scratch and be fully unconstrained. And that actually takes you to the best outcome. we used to start from existing systems and it was shortcut it would save a lot of work but if that work is actually easy for a really smart model

there's no need to constrain it from starting from that point yeah I think it's amazing especially like from the sort of research what's possible angle but I'm also curious about so like you know when we have these systems like when we have WeVA Cloud I guess there's like there like a few levels to it it like we sort of have like the knowledge of how to host it like how to run it and like how to run it at scale and then like how to debug it And I think those kind of arguments I can see these systems certainly like having a huge impact on that if not completely being able to do that. But then there's something to, I think, like the concept of like managed infrastructure where it's like you have like a provider like Redis, as we've been mentioning, they have like Redis cloud. And so there's probably still something to like these systems that like, that still have a managed infrastructure somewhere else, but can you still tailor those? Does this research have some intersection with that kind of thinking?

Totally. I think that, yeah, look, there's business reasons you might want to have someone else host something for you. You just don't have people on call managing it. There's kind of like technical reasons, like maybe you don't have the team to go and build out or check or even write your specification correctly so you're going to pay someone. You know that Reddit is the right specification for the system that you've built and you're not a technical team or something. I still think that the idea that we're working on are eventually going to slot into even those systems because you could imagine a hosted version of something like SkySynth or SkyDiscover Optimize or Synthesize that given a streaming set of queries, great. There's still an interface of puts and gets, but given the last 24 hours of puts and gets, I should have a resynthesized system that's going to do really well on that distribution of traffic. so I actually think this will fold really well into that model yeah it's so cool because you might have like these machines with like a shared set of machines with their RAM

and disk and then it's like perfectly allocating what fits into it with this new one yeah wow that's so cool yeah amazing I guess maybe I'd love to kind of come up the abstraction quite a bit and maybe talking a little more broadly about like systems that everyday people might want to use. Alex and I had a little joke before we were, before we started recording where I'll explain to Alex that I use Zencaster for recording instead of Riverside. And we're talking, I think I know of some software that might be able to create me a podcast recording software. How do you guys think about like this idea of generating software for say note taking or email? I can chime in briefly on this, which is just to say that humans are much messier than machines. So to write out the semantics, the actual formal specification for a key value store, it's ugly. Specs are hard to write. We're trying to make that easier. We're doing research on that too.

But there's kind of formal languages to model interaction between machines. It's much trickier to kind of formally model. How would you even specify what a note-taking app should do, right? The way that we often do this, you know, there are people that actually do this. We have a name for them, and they're a product manager. Product managers are like, they're kind of responsible for translating between the conceptual idea of what we want to talking to engineers and trying to explain to them in their language, still in English, but in English that the programmers understand. And the programmers go and write it in code. So today, if you look at SkySynth, like just the SkyDiscover framework in general, as we think about specification, a lot of what we're reasoning about is either you give us a formal spec or you kind of, in human language, describe what you want, but we kind of generate tests that encode the human spec, the intent of the human, you could probably get to a world where you could just as well synthesize kind of sets of tests of behaviors with a note-taking app, where you can take notes, you can go and create a new note,

kind of feature descriptions or PRD-style designs. That's what a spec looks like for a product. I think that we as a Sky Discover team are really quite focused on the core systems aspect of this, partly because it's just a really hard problem. But I think that the product synthesis side will be really interesting, and just looking at what the right medium is to capture a sec for a product, which to me is a PRD, we're not looking at that in our research, but I do think some of the techniques we're developing would be applicable. Yes, yes, yeah. As Alex said, so Skydiscarious Synthesize has two main arms, either test-driven development or formally specified, like doing formal verification on formally specified systems. Yeah, unfortunately, it's incredibly hard to capture what humans actually desire from these unverifiable domains, like design me a good web application, a nice UI.

We've been also trying to build some UIs for some of these systems we built using Sky Discover, So that part takes from other set of skills, definitely. Yeah, that kind of reminds me of what Shu was explaining earlier with the, like, I think, like, the idea of asking someone, like, what recovery can you tolerate? Like, you know, I've done a lot of talking to startups, entrepreneurs, like product builders who are, like, this idea of communicating these ideas. I think there's like a huge gap there that's really interesting and just, yeah, what specific, like when is the specification like a chat bot now? Yeah. Yeah, so at the start of our framework, there is a spec building part. The spec building part takes in some specifications from the human about what should be built.

Now, for the case that is test-driven development, the framework essentially checks if the spec that the human provided is clear and comprehensive enough. In that case, it will start doing some profiling on the workload that you specified. It would go and see what other specs exist in related code bases that are out there, et cetera. But there are also many cases, like you mentioned, where the input specs are not clear and human input is needed. Then, yeah, it acts like a chatbot in the sense that it tries to iteratively disambiguate what's going on with the human. Amazing. Yeah, so I know we're kind of coming up with a little time, and I usually anchor the WBA podcast with, you know, just generally, like, what excites you about AI. but I had to ask one more technical question about this.

How do you think about optimizing the systems? Is it something like, say, like GEPA for system design? Or is it like it? Yeah. I think optimizing systems, first of all, the problem is larger in scope because systems, if you care about a system, generally that system is also very large. And even finding out what is the right way to optimize, what is the target in it meaning a code base is going to be huge there are like hundreds of things that you can optimize but there are specific things that are worth optimizing and in optimizing systems there also there's also one more point unlike say text optimization that the evaluation is not that easy like if you really care about say hardware design or if you care about something like even inference engines you optimize something and the loop of getting the full feedback is like running lots of tests. If you care about VRLM, there's like probably 25,000 or more tests. And then you actually are going to run all the workloads to check for correctness. And then you

will run several workloads, like different prefills, different decodes, different agentic workloads to get some numbers in terms of say throughput or whatever. And again, even though in text optimization, also there is generally a multi-objective space with trade-offs. In the case of or in the world of system optimization, the trade-off is actually very real because you tune something for memory your throughput will be fully bad or or you are tuning something for latency and your throughput is just going away so it's very hard to and and as always it's very hard to define these trade-offs it's not like the human knows what is the right perfect point to have so in optimizing of systems which is obviously like one of the arm that system optimization had been a thing all the laws had been there for how you find the right problems to solve and how you optimize them in the sky discover pipeline making it agent again the next version which we are hopefully planning to have is where the agents can themselves scout for right problems the agents can themselves build a small simulators you can imagine or some sort of like

proxy evaluators which can give it the capability to help them so once it is able to hill climb then it becomes a hill climbing for problem which is equally same as like jepa for text optimization or any other technique which could be doing evolution over the text. But the problem becomes identifying the problem and getting the feedback in a way that the LLM can do. If I if I like give you a number in our KV generation like the key value generation or optimization probably 80 to 90 percent of the time is spent in evaluations. Hardly 10 percent is actually like text manipulation. So, the bottleneck becomes that and if you want to scale your agents now if you want to like run hundreds of agents doing this, you your objective actually becomes like a meta problem which is first get me the right kernel that is supposed to be optimized and get me a right proxy to actually measure just that kernel which is like very fine grained details or fine grained profiling that is going to help me in identifying how to evaluate and how to find out the bottlenecks which are the key to getting better performance.

So that's how we kind of see that it's like a meta problem and which doesn't really exist in the text space because in text space it's more, again, it's not a simpler problem. I'm not trying to infer that. But it's more defined in that world and the bottlenecks are different. So also circling back to what Alex said at the beginning of the podcast, actually, about, so given some evaluators, you could imagine trying to optimize some prompt of a system or a function that is doing a specific job as a select part of the system. But now, if you start from the specs, it's not even clear what the tests should be. It's not even clear which parts of the code base should you be optimizing. it's moreover it's not even clear if the specs are complete

did you the user specify maybe some workloads but as you said hopefully the design system should be robust it should generalize the perturbations around what the user asks for so can you come up with some other adversarial workloads to test the system and moreover, what if the specs change over time? Can you adapt to that sort of outer loop and get human feedback to iterate on your tests even more these sort of workloads over which you optimize more? So, yeah, it gets a little broader than the sort of classical evolutionary systems with which we use to optimize functions, et cetera. yeah i guess if i can just give my quick take on it i think of it as like gepa has this um pareto frontier of candidate program like you know prompts usually it's used

for text optimization but like it could be candidate programs like it could be code for the database designs and then the pareto frontier i guess instead of being like training examples and again like the text sense like you know particular prompt for writing a blog post and like for like, you know, this one prompts wrote a better blog post about mitosis than this other did about another topic in biology. And like, that's how you think about it in that world. But in this world, I can imagine that the tests like kind of define the Pareto frontier. But then I think what I'm what I'm what I agree a lot with this idea is that the evolutionary optimization should also be coming up with tests. Like it should be able to propose new tests and like evolve that concept as well. And evolutionary optimization is definitely a very cool topic. I think it's a fun thing to think about. Amazing. Well, Alex, Mert, Shu, Shavima, it's just been such a fun conversation. I've learned so much and so cool to meet you all. Thank you so much for joining the WBA podcast. Thank you so much. Thank you. Thank you. It was very fun to talk to you.