接続は認知ではない：マルチエージェントが静かに失敗する理由（Vijoy Pandey 博士）

Your AI agents can pass messages to each other flawlessly and still completely misunderstand one another. Today's guest has the data to prove it and an open source fix. Welcome to another episode of the Super Data Science Podcast. I'm your host, John Krohn. My returning guest today is Dr. Vijoy Pandey, SVP and General Manager, in other words, the head of OutShift, which is Cisco's incubation engine. On his third visit to the show, Bijoy reveals that every multi-agent session his team analyzed contained at least one hidden cognitive failure, and he explains the new open-source protocols OutShift built to keep agents aligned. He also details a record-breaking quantum network running on real New York City fiber and introduces new chips being built to compute on probabilities instead of zeros and ones. Enjoy. This episode of Super Data Science is made possible by Anthropic, Dell, NVIDIA, and Parallel. A quick production note before we get into the great conversation in this episode.

This episode was recorded before the Internet of Cognition software that we talked about throughout the episode was shipped. And between recording and release, some names changed. So OXP, which we refer to throughout the episode, actually ended up being called SCOPE, which is an acronym standing for System for Cognitive Observability, Provenance, and Explainability. Yeah, that's an acronym. But there are a couple protocols mentioned in the episode that are actually not acronyms. So those are ACCORD and CONCORD. Those are just protocol names. They are not acronyms. So a couple of clarifications for you. Now let's get into the conversation. DJ, welcome back to the Super Data Science Podcast. Such a treat to have you here. How you been doing? Well, it's been a treat for the past two times, and it's a treat again to be here. So thank you for having me. For sure. Two of my favorite ever on-air conversations. So delighted to have you back. We understand. It's a fast-moving space, and so you're going to have lots of updates for us.

Right now, everyone seems to be trying to make individual agents better. That's a lot of what we hear about. That's a lot of what we talk about on the show. But you, since you were last on the show nine months ago, you've been putting lots of multi-agent systems into production. What have you learned from those deployments? Let's start with where the agentic computing arc is, John. So if you think about agentic computing, it started with isolated agents. That's what you just mentioned. I think we went from isolated agents to islands of agents. where you can have many agents, sub-agents, all working together, but within the same harness. So within a Claude code or within Codex or within the harness of your choice. What we did was we took isolated or islands of agents and connected them. So we moved from isolated islands to connected agents.

And we've been deploying that over the past 18 months plus in a variety of verticals. So we've built multi-agent systems to solve for pain points and use cases in digital surveillance, in network configuration, in cloud platform engineering, in medical health and diagnostics. So a bunch of these use cases. And what we found is the need is massive, first and foremost. We are looking at agents that come from different vendors that sit on different clouds, that can access different tools with different identity and auth, and they belong to different organizations. So the need for connecting these multiple agents is actually massive. We've also seen a whole open source ecosystem appear along the way. So we ourselves formed the agency, and this was an open source ecosystem that's spelt AGNTCY, by the way.

And this open source ecosystem was created for agent discovery, identity, messaging, and observability. Since then, we've actually worked with A2A and MCP under the AIF. This is the Agentic AI Foundation. And we've also formed some other open source projects like AI Catalog with the Googles and OpenAIs of the world. So all of this has been happening to then connect agents. What we found out was, though, as we started connecting agents, we realized that agents are not coordinated. So just because you connected agents doesn't mean that they're coordinated agents. And that is leading to a whole set of failures that we are calling silent failures with loud implications. Because you can't measure them. You see the dashboards being green today, but the failures are there and your multisimating systems are producing wrong outcomes.

Yeah, this reminds me visually of a conversation that we had on the show a few weeks ago. So in episode 1030, I had Gaurav Patak on the show. and Gaurav was talking about how when humans disagree about some figure, you know, there's some important figure that, you know, you're going to be presenting to executive leadership. And so people in a meeting, you know, three different people come with three different opinions of what this figure should be. And you'll kind of, you'll have an argument and you'll figure it out and you'll come out of the meeting with either the answer of, okay, this was actually the right number to work with. or you come out with some action plan to figure out what the right figure is. But when you have three agents that come to some kind of virtual conversation in some embedding space with their figures, it can be a completely silent problem, like you're saying.

The process that unfolds, you probably have a lot more insight into this than I do, but it seems like we can just very frequently have that process, which should have been argumentative, just result in one figure being presented very confidently as opposed to trying to figure out what's really going on. So first of all, John, thank you for bringing that up because I think first and foremost, I need to go and talk to Gaurav and we need to start collaborating together. But one of the things that we've found in all of these vertical segments that we've deployed multi-agent systems in is exactly that, which is agents together can connect, but they cannot think together. And the other way to think about this is connection is not cognition. So there are a whole set of cognitive failures that happen when agents start exchanging meaning, when they start exchanging semantic information.

So I'll give you some other examples. Let's take a very simple example of an SRE agent and a SWE agent. These are things that we deal with all the time. When one of the agents says, let's roll back, an SRE agent might mean the last configuration. For the SWE agent, configs are not really the thing that they deal with on a day-to-day basis. So they might think of, let's roll back the entire software version. and so there's the word, the keyword rollback means different things to different agents because that's what they're accustomed to and we call this a grounding problem and take something even bigger like a protein folding specialist trying to propose a candidate but a specialist proposing a protein candidate is not the end all and the be all you have to take it to a wet lab agent that then runs the synthesis an agent that tests for efficacy. Then you have coordination with a regulatory agent

that is looking at what FDA can approve and what that proposal looks like or pathway looks like. You can then take it to a supply chain agent to see whether it's manufacturable. That happens all the time. Take an enterprise that we are so familiar with. You have a 2 million deal that is flowing through a deal desk flow and you have sales and legal and finance and product and marketing. And all of these people mean different things. When they say, Swiss will say the code is ready. That does not mean you can sign it up to the customer. And that does not mean that sales should be communicating that date to the customer. So what does ready mean is actually pretty fluid and ambiguous in these kinds of situations. So we've been looking at all of these cognitive failures and what we've been realizing is the system, an agentic system, can become more consistent over time through these cognitive failures without being more correct over time.

And these failures tend to compound. And we've done, of course, we've looked at multi-agent systems and all these vertical spaces that we've deployed them in. But we also took public traces from chat dev sessions, from MMLU probes that exist out there, and we analyzed them. And what we found was pretty surprising. We found that 100% of all multi-agent sessions had at least one cognitive failure. Oh, wow. We found that these failures compounded. So if there was one failure, there was still 98% accuracy in the multi-agent system output. You go to two failures and it drops to 83%. Now, you go to three failures, it drops to 16% accuracy. 16, 1, 6. 1, 6. That's as good as randomly picking an outcome.

And so these failures are, again, you don't see them because your observability dashboards are green, but these are silent failures that are going to impact the outcome that you're looking for from a multi-agent system. I want to hear more about these tests and how you evaluated them in a second. I want to clarify quickly for our listeners what SWE and SRI is. probably most listeners are aware, but SWE is software engineer. It's a TLA, a three-letter acronym for software engineer. And then SWE is SRE, site reliability engineer. And the key point that Vijoy is making there is that to those two different kinds of personas, a software engineer and a site reliability engineer, if you think about those as human people, then those two different people would have different ideas in mind of what the word rollback means. And so similarly, if you create an agent that is specialized to be a software engineer, a SWE, or a site reliability engineer, a SRI, the context that you provide them with,

the expectations for them would be different for what rollover means for those agents as well. And I think one of the key things here is if you had a software engineer and a site reliability engineer as humans, kind of similar to my example earlier about three people disagreeing over a figure, you are likely, if you have the SWE and the SWE humans in a meeting and they started talking about rollback, one of them would probably say, you know, a more experienced SWE or SWE would say, wait a second, I think we might be talking about different kinds of rollback here. Let's make sure that we're on the same page and we have the definition right. And I think what has happened in the quick evolution of multi-agent systems is that we're not getting the place for those cognitive clarifications to happen, and so cognitive failures occur. That is correct. And I think the way to think about this is we've connected agents, so we've built the infrastructure where agents can now talk to each other

and pass perfectly valid, accurate messages, but they don't align on what those messages mean. And that is the next step in the evolution. So, yes, things are happening quickly, but it also means that there is a lot more work to be done. It's not that we throw away the stack that exists. We need to build upon the stack that exists. So we have isolated agents. We built the stack to now connect agents. Now we need to build a stack that can exchange meaning, that can exchange cognition state. So it's a perfectly fine point in that evolutionary journey for agents as well. Yeah, it's the only way we can go. So it's terra incognita for us all as we figure out how to do this. Tell us a bit more about the evaluations. So when you were seeing those situations where when there's three cognitive failures in understanding between different agents, you're getting that 16% accuracy, how do you conduct those tests? I guess that's something that you're doing at agency.

Again, that's A-G-N-T-C-Y. Right. So right now, agency and A2A and MCP and the entire ecosystem that exists out there has been built to connect agents. And when you connect agents, yes, we do have a whole observability stack that we actually worked with Microsoft and pushed it on OpenTelemetry about October of last year. So a year ago, roughly. That open telemetry observability stack actually tells you whether agents are working well, whether their uptime is okay, whether their reasoning engines are functioning properly. What it does not tell you is what happens when these two agents start exchanging messages and when they start exchanging meaning. So you can still bring in connected agents that can have different goals, that can have different contexts and assumptions and constraints and beliefs. And that was some of the examples that we saw earlier. And when you bring these agents together into a multi-agent system, what you have is a lack of grounding. What you

have is a lack of alignment on terminology. You need to discover terms that you don't understand. You need to resolve conflict that you disagree upon. Then once you have aligned on intent, The next step is to then coordinate, just like human teams. So once you've aligned on the goal, that is, let's go ahead and solve this problem together. Let's build this drug and this percent efficacy to pass FDA scrutiny and then be manufacturable. Let's manufacture that drug together. So that's the goal. Once you start working on that problem, just like human teams, you need to coordinate and negotiate because things are not going to be perfect. and every agent comes in with their own local optimum. Every agent comes in with their own local incentives. So when you have this coordination and negotiation problem, every agent needs to compromise a bit and then work towards a global optimum.

So that's the coordination and negotiation problem. That's the second step. And then finally, once you've achieved a goal, is there a verification layer? again, at the cognition semantic layer is the verification that tells us that you've done your job well. It's almost like all of us going through a performance review at the end of a financial year and getting a score as to how we've done. So we need to do that again at the semantic knowledge cognition layer. So those are the three steps. The shared intent, grounding, resolution discovery, then the coordination and negotiation to actually go through that task and achieve that intent. And finally, the verification step to ensure that we've met those goals at the cognition and somatic layer. So that's the stack that we're building out now. We're calling it the Internet of Cognition. And it's something amazing because it actually

follows what's happened in human intelligence right to the dot, where human intelligence also started out through individual intelligence getting better over hundreds of thousands of years. Then roughly 70,000 years ago, there was the invention of language which caused a cognitive revolution. And cognitive revolution pivoted us from individual intelligence to collective intelligence. And that's exactly where we are with agentic workflows, where agents are getting bigger and smarter over time. And we see that all the time, but they're still isolated. They're just now getting connected through the workflow agency and A2A and MCP. And now we are building the Internet of Cognition to move towards collective intelligence, where we are inventing protocols, where we are inventing shared knowledge, where we are inventing ways for

agents to then share reasoning together. Yes. So we're trying to have the same kind of explosion in collective intelligence facilitated by a common language. Now, with agents like we got with humans 70,000 years ago, it seems to me like one of the special things about finding that common language in the way that you're doing it at agency is that, correct me if I'm wrong, you're trying to make sure that that language really is common across whatever kind of agentic framework is being brought to the table. So first and foremost, agency is an open source stack along with A2A and MCP that allows agents to connect together. And the good thing that has happened over the past, I would say, nine months since we last talked is that there is actually convergence across all of these open source projects that have popped up to solve for the connectivity layer, which is a really hard thing in the industry,

as you can imagine. Convergence between humans is still a hard problem. And so the fact that agency and A2A and MCP, they all sit within the Linux Foundation and they're all working together on a common infrastructure stack so that we can discover agents in a common way. We can describe agents in a common way, that's the AI catalog work. Then we can discover agents in a common way. We can then attribute identity and access controls to agents in a common way and then communicate whether it agents to tool using MCP or agent to agent using A2A using a common messaging layer which we call slim or secure low latency messaging, interactive messaging. That stack is becoming common and integrated. So that's a good thing, first and foremost. But what's still missing is that cognition layer.

What's still missing is that semantic layer. And what we realized is that the entirety of the networking infrastructure for the past 40 plus years was built for connecting data endpoints. And now we have a new kind of endpoints that are starting to think. These are intelligence endpoints. And so we need a new layer of protocols that connect intelligence endpoints that looks at meaning and not just data as a blob. And that's what we are building. We published a paper on this late last year around online protocols, but we are actually building those protocols right now. Excellent. Well, that all sounds exciting, Vijay, but how much of this is really shipping and something that my listeners can use as opposed to being another white paper? Well, we just released an entire stack that allows you to look at first silent failures

in your multi-agent systems, to measure them, to characterize them as to what kind of a failure is it. Is it goal alignment? Is it coordination and negotiation? Is it a verification problem? So one of those three. And then we've released two protocol implementations to solve for two of those problems. So let me walk you through that. So first and foremost, think of this entire architecture in three layers. It's not a very abstract level. First and foremost, there are these protocol layers that get triggered when you see something going wrong. It's like you're driving a car and you have lane correction turned on. So when you're driving a car, you're all good, you're running down the highway, no problems. But then when you start seeing, when the car starts seeing that you are digressing, you are moving into the other lane, maybe you are sleeping a little bit or whatever it is, I don't know, I don't recommend doing that, pull over and sleep.

But there might be something, maybe you are chatting with your kids or they are creating a ruckus. Or the car starts veering into the next lane or maybe there is construction overload or who knows. So when it starts veering into the next lane, your car then triggers this feedback loop to you saying, get back within the lane. We think of this entire protocol layer in the same way. So there is an observability and explainability platform called OXP, observability and explainability platform. What it's doing is it's measuring all of the traces, it's looking at all of the traces flowing in your multi-agent system. And it's doing this through the open telemetry extensions that we've already shipped last year that I talked about. So it's sitting in agency, it's sitting in hotel. You can use it, it's de facto standard open source. And it uses those traces in this multi-agent system to constantly look at what's happening at the semantic layer.

And it's looking at cognition drift. So whether it's alignment drift, whether it's coordination drift, whether it's verification drift. So any one of those failures that we talked about, it's looking for signals of that. So first and foremost, you can just use OXP and measure cognition failures in your environment. That's the simplest thing you can do. and you will see similar results that we saw. And so characterize them, look at what's happening. But then what we do is this OXP platform, as it measures drift, it then triggers some protocols to fix that drift, just like the lane change. So when it sees intent drift, before agents even start working on a problem, it can detect drift in intent. that these guys are not even aligned in what they're supposed to do. We trigger a protocol called ACCORD. It's an acronym. I'm not going to say what it means.

I'm sure I'm going to butcher it out. But it's an acronym that actually allows for intent to be aligned. And this is a four-phase protocol that produces a locked contract for your intent. It's a shared artifact that all agents then commit to before any work can start. So that's the first protocol that we can trigger. And that's available right now. And that ACCORD protocol is spelled A-C-C-O-R-D. That's correct. And it's A-C-C-O-R-D, and it's an acronym, and you can find it. And you can read the acronym yourself. But it's basically locking the intent. And it's a shared artifact across all of these agents pre-work. So that's the first one. Then once work starts, again, your OXP platform is looking at all of these traces and what's happening between these agents and the semantic layer. And if you see consensus drift, or if you see coordination getting drift

or getting in failure mode, then we trigger another protocol, and that's CONCORD, and that's C-O-N-C-O-R-D. That's also an acronym. and what it does, it's a consensus protocol. So it resolves decision conflicts, or it resolves divergence during a life task, and again, brings the agents back into the lane. So that's the Concord protocol. So these are the two protocols we've actually released so far. You can imagine us releasing other protocols that look at other kinds of cognitive failures in the future. and the one thing to keep in mind again is these get triggered only when you steer across your lane. In regular traffic, you're doing fine. You're just running using standard GRPC or MCP or A2A, the protocol of your choice. That's what you're using. That's how agents are communicating. We don't touch them.

We only measure them and when we see these drifts and failures, we trigger these protocols. they bring you back in line and then they get out of the way as well. I love it. So it doesn't matter how I create my agents or how me and a bunch of different teams create different agents using different frameworks for each of those different agents and then we can bring them together. We can coordinate together on getting the right intent with Accord and then making sure that all the agents are staying aligned with Concord and all of this is part of what you call OXP, the observability and explainability platform. Did I kind of summarize that correctly? All these are actually triggered by the observability and explainability platform. So the platform itself is almost like sitting on the side looking at what you're doing. It's like your auto navigation in your car is looking at what you're doing. It's actually not driving the car. But as you see the car do things it should not be doing,

steering into a different lane or getting too close to the car in front of you, it then triggers a protocol to then react and get you to safety or within your swim lane. And then you're back to driving the car. So that's the way to think about this. And at the bare minimum layer, you can use OXP to just characterize all of these cognition failures in your multi-agent systems. So even if you don't want to, I don't know why you wouldn't want to align them as a next step. But even if you don't want to align them, you can just observe and measure and characterize your multi-agent system at the cognition layer. So, that's already pretty useful. And by the way, one more thing on how you can use them, because it is a pretty heterogeneous world out there and there are agents that are coming in from different harnesses that do sit on different clouds and so the heterogeneity in this environment is pretty massive. So we've tried to make usage of OXP as well as these protocols pretty straightforward.

So OXP, like I described, is based on OpenTelemetry, which is the de facto telemetry stack for any cloud-native application. So if you're using any cloud-native stack, Kubernetes, whatever, you will get OXP as an extension to that period. because we are looking at OTEL traces and we are building the semantic layer ourselves just by looking at the OTEL traces. So you don't need anything extra. So XP is by default through a cloud-native stack. The protocols themselves are available through what you prefer. So you can use a skill and deploy that skill in your agent, which is the least friction way of using these protocols. and they again sit on the sidelines till they get triggered. You can use a sidecar-based approach. So this is what we've done for a decade now, again, in a cloud-native stack. So an onboard-based sidecar, as an example,

or an eBPF-based sidecar. So there's a sidecar-based approach. But we also have bindings to some of these common lower-layer protocols. So there are bindings for A2A, there are bindings for MCP, their bindings for SLIM. This is an agency messaging protocol. So you can actually use Accord and Concord above NCP, above A2A, above SLIM. So pick your approach. We give you the least friction way of using Accord and Concord. And we'll be doing that for all the protocols that we ship. Amazing. And I am assuming, based on the way that most of the things that you and I have talked about in past episodes on this show, that all of this is an open standard. It's freely available for anyone, right? That's great that you brought it up. All of this is open source. So not just the open telemetry traces and extensions, which are open source last year. A2A, MCP, SLIM,

these things are part of the ecosystem today for connectivity. But Accord, Concord, as well as the engines that support measuring cognitive drift are all open source. so please use them please characterize your multi-agent systems give us feedback get involved in the community that's what we ask you spent weeks choosing the right model for your agent how long did you spend choosing its web search agents need reliable and accurate access to the web to do their best work and that's where Parallel comes in Parallel is web search built for agents not people and it delivers the best quality results at every price and latency tier Building a voice agent? Parallel Search Turbo answers in about 200 milliseconds for only a dollar per thousand requests. Tackling thorough research? Parallel Search Advanced digs deep to answer your toughest questions. Parallel works with any model and drops straight into Cloud Code, Cursor, Vercel, or Langchain. And it goes beyond search with web monitors, deep research, and batch record enrichment.

Get started with a free MCP plus $5 of free credits every month at Parallel.ai.sds. That's Parallel.ai.sds. And why is Cisco doing this? Why is Cisco doing so much, putting so much R&D time into developing these free open source opportunities, these free open source technologies? First and foremost, we are OutShift. Outrift, we are the incubation engine for Cisco. Our job is to look at where technology is emerging and then figure out how we can build products and projects in those emerging technology spaces, reduce tech risk and market risk for Cisco as Cisco tries to enter those market spaces. So that's what we do. And a big chunk of what we end up doing, because we are always exploring the frontier of technology, a large chunk of what we do is educate the market.

A large chunk of what we do is figure out what makes sense. So exploration is a big deal for us. And then as we explore products and projects in these spaces, we can figure out which problems matter to our customers so that when we do build products, yes, we are not a charity, we will be building businesses and products in these spaces. But we are gathering early signals and figuring out which problem spaces and which technology stacks actually matter for our customers, for the use cases that they are interested in. So we are so early that we are exploring, but soon we narrow it down. And yes, there will be products that Cisco will build that incorporate a lot of what we're doing. But also one of the things that we believe in is that open source is actually the right approach for tackling any technology problem. We are big believers in open source because open source is fundamentally advantageous

to every player in the ecosystem. Whether it's a vendor, whether it's a developer, an operator, whoever it happens to be, a cloud provider, an open source approach is fundamentally more more advantages and more profitable than having closed ecosystems and we are firm believers in that and we've always relied on open protocols, open standards open source to then build products and projects in those spaces I agree with you 100% I also am a big proponent of open source not only as something that's great for individuals, as you say, developers, the community at large, but ultimately it can entrench a particular solution and it allows products to be built around it. Countless big businesses like Databricks have been built around this idea of having something open source available and then

productized versions, maybe enterprise tiers with extra security, these kinds of things built on. and it makes so much sense to me. In fact, when this episode airs, I will have just returned from the Open Data Science Conference in San Francisco and that's my favorite conference because you're mostly focused on open source solutions to AI problems as opposed to locking in with any particular vendor option. Isn't it kind of convenient? Maybe it's just the way that you guys think at Cisco. You know, Cisco, when I think of that word in my head, it's in my embedding space in my brain, it's very closely related to just the idea of networking. I think that's what Cisco has been predominantly known for over the decades that Cisco has been around. And so it's, yeah, it's interesting or convenient or that you're focused right now at OutShift on these kinds of problems you've been discussing on the podcast so far.

And we have very different kinds of problems we're going to get to next in the episode. But on the problems you've been discussing so far, it's been all about networking between agents. So that's kind of interesting, isn't it? I think of computing and scaling computing across three axes. You can scale computing across the axis of algorithmic improvements. You can scale computing using vertical scaling, which is building bigger and bigger of something, a particular something. but then you can also scale computing by horizontally scaling. So the three axes are algorithmic, vertical, and horizontal. And that's how computing has scaled over the past seven or eight decades or so. And we at Cisco, we feel that the horizontal scaling axis is the place where we play best. So yes, you can narrow it down and say it's networking, but it's a broader statement than that. we bring in the aspect of horizontal scale to a computing problem that others may not look at.

So others might be building bigger and bigger brains, in this case, bigger and bigger models, or bigger and bigger agents and smarter agents. But we are looking at scaling out superintelligence rather than scaling up superintelligence. You could be building bigger and bigger computers and servers. And that's what companies have been doing for decades. we invented the internet and we invented distributed computing and we are a large player in the cloud computing space because we allow for distributed computing to take shape through the networks that we build. So our thesis has always been to build secure observable networks that take computing and horizontally scale it. We do work with vendors who are doing the algorithmic scaling and who are doing the vertical scaling, but it's our job to then horizontally scale that computing paradigm. So yes, it might seem convenient, but that's where we shine

and that's what we think are the problems that we're best suited to solve. Makes sense as a focus, for sure. With you using so many driving analogies as we've been talking about this topic, you know, correcting a vehicle as it seems to drift into a lane. I love that analogy. makes a lot of sense as we think about agents working together and drifting away from that problem. But at the time of us recording, and this might be outdated by the time that this episode airs in about a month's time, but at the time of recording, I just became aware this week about a really cool new benchmark called Driving Bench I don know if you heard of this but it a benchmark where they take general cognition models So Claude Fable 5 ROC 4 GPT Astra and they have it in real time try to do a simple driving course through a parking lot where it's just real time video feed

input the uh the general cognition llm has access to uh steering control gas brake and pretty interesting so the results show that uh so grok 4.6 for example they give each algorithm three tries in this in one continuous chat and grok 4.6 it the best that it scored across all three tries was it got 11% of the course, which is like not even making the first turn. Same thing for GPT 5.6 Sol, but Claude Fable 5.1, a new bigger class of algorithms. I guess it's like a vertical scaling example. Claude Fable 5.1 on its third try gets halfway through the course. GPT 6 Astra on its first try got halfway through the course. On its second try completed the whole thing. and it's a pretty wild it's hard to kind of wrap my head around how that can even work how a model that is trained you know there is a vision component to these models that's critical

obviously to being able to drive but that they're able to learn how to do something like driving in just a few shots, presumably without much or any explicit training on that kind of task. It's pretty wild. Yeah, and so I haven't looked at the benchmark yet, but is this, I'm assuming it's on a 2D map and you're staring through a 2D map, or is it, I mean, I don't think they're actually driving a 3D, like a car in an actual world. They're driving, they're driving a car, a Toyota Corolla. That's pretty amazing. Because I think it's, I mean, that to me is also the next frontier where we're sort of reaching the capability curve on a 2D space. But getting into world models and where that direction is headed towards, that is the next frontier. And even there, like we were talking about earlier, yes, these models will get really, really good at individually steering a car or an embodied AI physical robot.

But then how do these swarms or societies of physical embodiments work together to complete a task is actually going to be really interesting because you cannot have a singular entity do everything. So you can't have a singular robot. I mean, at some point, that physical AI needs to transfer or do a handoff to a different entity. And whether it's handing off a box or whether it's handing off an incomplete picture of a car or something or the other, there needs to be a handoff between many, many, many physical embodied agents, robotics. And it's the same problem yet again. Because if you see these cognition failures in software systems, and they could be silent with loud consequences, I can guarantee you that these failures will not be stagnant and will be really loud and scary when it comes to physical embodiment in the real world.

Yeah, you touched on exactly what's most surprising about this, which is, you know, without having seen the research, and I'm not sharing my screen with you right, because you can't see, you don't have the benefit of seeing the Toyota driving around, but it makes perfect sense to think that this would be some kind of simulation, some simplification, because it's kind of, that's what's surprising about this to me, is that without this kind of explicit 3D world model training built in, this latest class of LLMs, Fable 5.1, GPT-6 Astra, were able to kind of, the world models are inferred and can kind of be represented within the embedding layers of these models that were trained on still images and text from the internet. It's pretty wild. And actually brand new news at the time of us recording this episode, at the time of recording this episode, is that Fei-Fei Li's World Labs, This business named World Labs has just been acquired by AMD for $8 billion.

And it's obviously, as you're saying, there's so much more to still be done in the physical world. But it is interesting that the scale of these models that weren't even designed to be able to navigate the world can figure it out a little bit now in some cases. Right. And I think the other interesting problem, again, to extrapolate, is how can you then take these world models, take these text models or video models, and then, again, align them, make them coordinate, go after shared intent, and then solve for business outcomes that you and I do every day long. And what does that look like in a year or two years from now? And are we building the right infrastructure? because one of the coolest things that makes agents, whether they are software or embodied, different from humans is that humans don't work on deterministic infrastructure,

but agents still work on deterministic infrastructure. And so there is codification of rules and regulations and what's possible and what's not possible in software that you can do to then make these agents collaborate safely, coordinate, negotiate, and then work towards a common goal, which humans, it's not that easy to do. So the fact that they are software is still a benefit, is a huge benefit, because we are not software and it's very hard to make us align on incentives. It might be easier to align agents on incentives. We sure hope so. And it's interesting you touching there around probabilistic and deterministic. It's a really nice segue into the exciting topic that I want to dig into for the remainder of this episode, which is around the computing substrate itself and how computing today is largely, I mean, there's, you know, what percentage of computing is?

is done with non-classical systems today. It's negligibly small, but you do have some interesting examples about things like quantum computing that we'll talk about in a moment. But if we think about classical computing, which is today 99.99999% of the compute that's happening on the planet, it is deterministic, which is interesting because we are increasingly in a probabilistic computing world from a software perspective. So when you and I were talking about topics to discuss today, that is, you know, you saying that to me yesterday as we were preparing for recording today, it's such an obvious thing. It's such an obvious mismatch that had never occurred to me. So fill us in more on this idea of OutShift, the group that you run at Cisco that's so forward-looking. You're trying to prepare us, as you mentioned earlier in this episode, for what's coming years from now. And it seems like one of the big opportunities coming years from now is to have probabilistic hardware to match probabilistic software.

That's right. Right now, computing is going through a paradigm shift from applications all the way to infrastructure. And what happened and that paradigm shift is the movement away from deterministic systems to probabilistic engines. that shift actually happened in the workload at the application layer first. And we're seeing that all day long. So everything that you're looking at in terms of agentic computing is actually a probabilistic engine. Your inputs are non-deterministic in nature. Give me the best or optimal answer. And what comes back is an optimal or a best answer with a certain confidence attached to it. So there's no binary here. So the workload moved towards the probabilistic engine, but the substrate that is doing the actual computation is still binary. It's still very deterministic. It's still matrix multiplications over GPUs. So there's a little bit of a mismatch between what's happening at the workload layer and what's happening at the substrate layer.

So it's almost as if the workload layer has gone to a dimmer switch where you can dimmer switch between complete darkness to complete brightness, whereas the compute layer is still an on-off switch, which is either it's completely bright or it's completely dark. There's no something in between and there's no distribution in between. So we have to fix that gap because if we don't fix that gap, we will always live in this world of approximations, and approximations cost energy, money, size. And so that's where we are. So there's a whole new set of computing paradigms that are emerging to build computing as a non-deterministic engine or a probabilistic engine. And one manifestation of that is quantum computing. There are many other manifestations of how you can build non-deterministic compute.

Claude Cowork now runs in the cloud, and that's changed how I work. Recently, I asked Cowork to pull my scattered notes from Gmail and Google Drive into a research brief for an upcoming podcast guest. Then I closed my laptop and headed out. I checked in from my phone on the subway, and Claude was still going strong. By the time I got home, a polished, well-sighted brief was waiting for me. claude is the ai for problem solvers it's the collaborator that understands your entire workflow and thinks with you not for you whether you're debugging code at midnight building a financial model or strategizing your next business move claude extends your thinking to tackle the problems that matter ready to have an ai that can tackle real work try claude co-work today on any pay plan at claude.ai super data that's claude.ai super data and check out claude pro which includes access to all the features mentioned in today's episode. Cloud.ai slash superdata. Love it. So let's talk about some of these specific

implementations. So the one that I've already mentioned is quantum. I think that's something that when we think about distributions that we can be working over when we think about having that and I love that analogy. It's so brilliant to think of it as a dimmer switch where we have this gradient from completely dark to completely bright. Yeah, so quantum computing is one of them. Why don't we talk about that first? Then there's other types of probabilistic computing that I'd never even heard of before you were telling me about yesterday. So if you think about probabilistic engines at the compute layer, the idea for this can be credited back to Richard Feynman in 1981, where, and he's a famous physicist. He's one of my favorite physicists because he always ran with good analogies and good statements. I think he's a scientist that every scientist wants to be. I think he's just the best. He just aligns with what you would want to hear

and he just grabs you and brings you in. So one of the things that when he was looking at these really large problems, and what do I mean by large problems? These are problems that really matter to humanity. So we see a lot of these, drug discovery, new materials discovery, weather forecasting, financial modeling, logistics planning. These are really large problems and what large problems have in common is that they run on really large spaces, state spaces. And the entire act of finding the best drug or the best material is actually a search algorithm that needs to run on this pretty massive state space. So that is what we are doing all day long And so all the problems that matter to humanity Can be boiled down to a search problem Across a really large state space

Now, to do these searches You can do the classical route that we just talked about But what Feynman said was A lot of these problems are nature problems And nature isn't classical Nature is quantum So why are we trying to put a square, a round peg in a square hole? Let's use quantum principles and then build quantum compute to then solve for these nature class problems. And so that was the genesis of the entire quantum computing thesis and we've been building that out. So we've been building computers that deal with qubits or quantum bits versus bits that we are familiar with. So a bit can be a zero or a one at any given point in time. A qubit is a zero and one at any given point in time. So the Schrodinger cat is both dead and alive at the same time because it's inside the

box and you don't know whether it's dead or alive. So it is both dead and alive until you open the box. Whereas a classical computer is a dead cat or an alive cat. We're hoping it's an alive cat. It's just an analogy, people. It's just an analogy. Don't get too attached to it. So qubits are useful because since they can be 0 and 1, they allow you to explore this vast state space search very, very efficiently and quickly. So they can be really effective in finding new drugs, new materials, new weather forecasting, all of the problems that we talked about. The problem that we face today is we are still a little early in that journey. So for all of these big problems, not just to meet the bar, but to be actually pragmatic and practical at solving these problems, we need about 100,000 to a million

cubits, physical cubits. What we have available today, the best case is about a thousand physical cubits. So there's a massive gap between the cubits we have and the cubits we need. Yeah, two or three or four orders of magnitude. Three to four orders of magnitude, but as these quantum compute companies like the Googles and the IBMs and the Atom Computing, Infection, Qera, there's a whole bunch of these. As they build bigger and bigger quantum nodes, that's vertical scaling. As the software people in quantum algorithms do algorithmic scaling, we, again, at Cisco, step in and say there is a path that is not being explored and that is horizontal scaling. So can we connect 100 of these 1,000 qubit computers and create a distributed quantum environment of 100,000 qubits?

So can we build a quantum network that operates in the quantum realm to then be able to create a holistic, logical quantum computer of 100,000 qubits? And that is what we are trying to do at Cisco. We are building a quantum network that can shift or switch quantum information between any pair of nodes without actually looking at or measuring that quantum information because that defeats the purpose. So it's the horizontal scaling from Cisco again. That's right. And one of the things I would say is the entire, again, this is where networks are dramatically different. because networks for the past 40 years have been built on one principle which is store and forward which is I store a packet that comes in I look at a packet header decide where that packet needs to go

and then I forward that packet to the next destination that's how the entirety of the internet has been built guess what, in quantum networking you cannot look at anything So there is no header to look at. There's nowhere to store the packet, look at it, and then decide where to send it. So that decision of where to send that qubit of information has to be done without measuring or looking at that qubit of information. And that's the challenge. And that's what we're solving for in our shift. Really cool. And just like earlier in the conversation when I was saying how much of this is just a white paper thing when we're talking about these agent protocols versus something that my listeners can actually use. Well, in this example, I guess my listeners aren't going to be able to be doing quantum networking from home immediately after listening to this episode. But at least there is something really happening here. So I understand that this idea of being able to have communication between quantum nodes is really happening.

There's a Brooklyn-Manhattan deployment. So I guess I actually live on the side of Manhattan just looking out over Brooklyn. And so it could be happening near me. Think of it as a separate kind of Manhattan experiment, John. It is a safer and better kind. But the way we've – so at Cisco, we've been building the entire quantum networking stack from the ground up, from chips to protocols to controllers to applications. and we've been releasing the chips separately we been releasing the protocol separately and we been releasing the application separately And we been doing that over the past couple of years We released an entanglement source. We've released a quantum switch just a couple of months ago. We've released a whole bunch of protocols, and these protocols might sound science fiction-y, but they're actually true. They are protocols such as entanglement distribution. They're protocols such as Teleportation detection sounds like Star Trek, but they're actually true.

And you can use all of these to build a quantum network today. Now, quantum networks have been built in lab environments for a long, long, long time. What we decided to do with some of our partners, QNECT is one of the partners, QTD system that supplies fiber in New York City is one of the other partners, and then New York University is the third partner. So along with all these partners, we decided to do a real world quantum network experiment. And we did that, I think, it was February of this year, if I'm not mistaken. This experiment was done over 17.6 kilometers, roughly 11 miles, from Manhattan to Brooklyn and back. And it was done on real New York City fiber. the dirty, humid, hot, shaky, going in subway stations, below buildings, below streets, that kind.

Hey, don't talk about my fiber like that here, huh? It's real and it's true and it's amazing. And so think of these fibers as real world fibers. They are real world fibers. and what we did is we said, let's run our network across 11 miles through 60 Hudson. This is the telecom carrier hotel for those of you who are not familiar with carrier hotels. This is where a lot of the fiber from across the Atlantic and from across the US all converge. So 60 Hudson is one of the busiest carrier hotels in the world. And this fiber actually ran through 60 Hudson. And what we showed is entanglement swapping, which is the basic bitwise operation, if you can think of it that way, in the quantum world. So entanglement swapping across these two ends, and we showed rates that are 5,000 times better,

5,000 times better than what has ever been done, even in a lab environment. So this network is real. It's in a New York City environment through existing fiber performing 5,000 times better than what you've ever seen in a lab environment. and the reason it could do that is because we are building this coherent stack all the way from chips to controllers and protocols, the software, and the application all working in concert, not just with Cisco chips and hardware, but also with our partner hardware like QNX hardware and working on existing fiber like QDD systems. So we are building the quantum internet one link at a time And this was the proof point in real life of at least 11 miles of a possible quantum internet. So this is going to be a really dumb question from a quantum newbie.

But I thought the whole idea of entanglement is that we could be, you know, kind of any distance apart and that entanglement is still in place. Yet in this situation, you're describing a physical, dirty, humid New York network cable that is somehow supporting entanglement? That's a great question because the way the entire quantum networking paradigm works is, the way I would describe it is, let's say, John, you're sitting in Manhattan. I'm sitting in San Jose. At some point, we need to exchange quantum state. so there's not actual classical information because that is going to break a bunch of physics laws so we're not really sending classical information faster than the speed of light there is still the no cloning theorem and it still holds we have not crossed that boundary because that would be tough but what we can do is we can take coordinated action

and exchange quantum state between the two of us And the way that works is there needs to be a common source, almost like an Amazon. Let's say there's an Amazon that sends a physical box to you and to me. This is the generation of an entangled pair of photons. One of the pair goes to you, one of the pair comes to me. So that is still very physical. So it is sent over these existing fiber links and it follows the speed of light principles. So that needs to happen. But once you get the box and I get the box, one photon each in this case, then the entanglement property can be teleported at distance. So now the property, the quantum information that sits in that entangled photon is actually an instantaneous piece of state information that is coordinated across the two of us.

So that is the teleportation part. But physically, you need to send these entangled photons. And then after that, the state information is a teleportation problem. So that's the way to think about this. So as long as you and I stay in the quantum domain, think of it as like that movie. Ant-Man, I think, is that movie? That's a kid. That's like a Marvel comedy. Yeah. So we've been shrunk to the quantum state. And when we are in that quantum realm, at that realm, we can exchange information using teleportation. But if we want to come back to reality, to the real world, the classical world, then we need to come back and we will not be exchanging information. We will be bound by the speed of light. So that's the way to think about it. But as long as we are within the quantum realm, the quantum network treats all of these nodes and makes them behave as one singular entity.

Hey, hey, this is your host, John Crone. In addition to hosting this podcast, did you know that I run an AI consulting firm called Y-Carrot? Yes, that's the letter Y and the deliciously crunchy veggie. at YCaret our team pairs decades of ml and software engineering experience with recognized expertise in the latest ai techniques such as all the key generative and agentic approaches from problem scoping and proof of concept through to high volume production deployments we've got you covered at every stage of the ai project life cycle to learn more head to ycaret.com from there you can click partner with us to give us some context on how we can help We're looking forward to hearing from you. Again, that's Y-C-A-R-R-O-T dot com. I can't candidly wrap my classical mind around teleportation or quantum. No, I can. In fact, even Einstein couldn't, and he called it spooky action at a distance.

So you and I are just mad mortals, and it's fine. not as smart as Einstein or smarter than Einstein and mathematically things work and in the lab they work so we take it, we prove it and we run with it alright, well I guess I can entrust an LLM to do it for me perfect, so that's quantum computing which is a non-deterministic kind of computing substrate that I'm sure every listener has heard of but tell us about some more niche ones that are also coming online. When Richard Feynman made the comment that we need to leverage quantum, because nature is not classical, nature is quantum, so let's leverage quantum computing to solve for these really large state space search problems. He took a leap, which is almost the right hand side of that spectrum, so it's the right bookend. Let's leverage quantum physics to then solve for these state-space search problems.

We all know that there are other kinds of physics. There is classical physics. And so there is this spectrum in between, between classical deterministic binary systems and quantum systems that are classical physics-based, that can also carry distributions instead of carrying binary data and can solve for these search problems in vast state spaces. And so we are seeing a whole landscape of startups emerging to solve this problem. And they are leveraging classical physics laws to build silicon that does computation on distributions rather than zeros and ones. So, if you think about the computing spectrum, you have deterministic systems working on zeros and ones, you have quantum systems working on superposition of zero and one, and then

you have the spectrum of computing in between that works on distributions between zero and one. And so, some examples of that, to just wrap our heads around it, are thermodynamic based systems. There is a company called Extropic. It's building these silicon systems. It's actually shipping them now that work on thermodynamic principles to then solve for an outcome. And the way they work is they find the low energy state in a thermodynamic system. And when that system finds that low energy state, that is the outcome that you're looking for. So imagine you're dropping a marble in a valley punctuated by hills. And so when you drop a marble, it settles into one of these valleys. And wherever it settles is the outcome that you're looking for.

It may not be the global outcome, but it could be a local optimal. So that is thermodynamic-based computing. But there are other types of physical laws-based computing. One pretty well-known startup that has emerged in the past six months is Unconventional AI. They've raised a lot of money to solve for silicon that works on oscillator-based principles. So unlike thermodynamic-based principles where the result is settling in on an energy minimum, So, here, these are oscillation-based compute elements where once they align to a particular oscillation pattern, that is your outcome. So, think of this as metronomes all swinging, and once they all align on a particular pattern of swinging, that's where your answer lies. So again, these are not binary systems.

They will all give you local optima, but that's exactly what you're looking for in these really large state spaces. Because you're looking for a local optimum, and that's where your current answer is. If you had more time, you could look for a better local optimum. But you may never get to a global optimum because there may never be a global optimum. There is never the best drug. It's just the best drug for the time being. it will always improve as our technologies improve and as our understanding improves. So you're always looking for the best outcome given the constraints of time resources that you have at your disposal. And that's what these systems are doing over a distribution, whether it's thermodynamics or oscillation-based or over superposition in quantum sense that are different from the binary deterministic world that we're so familiar with. Really interesting. So, yeah, so thermodynamics based, oscillation based, quantum based, three different emerging possibilities for non deterministic computing to match the non deterministic software that has become so prevalent with AI and large language models.

something to keep an eye on for sure and a really fascinating conversation for our listeners, Vijay, who want to be able to get more of your thoughts. I understand you have a kind of a reading recommendation for us that might help us out. First and foremost, the thesis that inspired us to look at this whole arc of human intelligence and then map it to the parallel that's happening in agentic evolution and towards super intelligence. There's work by Michael Tomasello. It's a pretty heavy read, but you have LLMs at your disposal. They can summarize, make it easier to digest. Go ahead and do that because the work that Michael has done on the evolution of human intelligence and what we can learn from that is just fascinating. And you will see those parallels play out today, every day. And it's not just me and Altshift that got inspired by it, but many, many academics, many, many frontier lab

people, many labs in these large vendor environments, they're all getting inspired by that work. So it's something to keep a close eye on. I myself have pivoted from reading books to reading research papers, which sounds pretty boring, but that's where the frontier is moving. So I read a lot of archived papers and I keep up with those papers out there. But I've been writing a lot about where the world is headed on my own substack, which is called coherent cognition. And the world actually talks about the cognition piece, the coherent piece, which is the quantum side and everything in between. So you can take a look at coherent cognition in Substack on LinkedIn, but also I'm starting a podcast on my own and the thesis behind that is to make all of this frontier research that is not available as easily as possible to you and I, to lay people who are trying to

understand and make sense of it, or from a business perspective, simplifying that for business people, simplifying that for infrastructure people, for software people. I'm trying to do that by talking to researchers who are at the forefront of deep technology research. So all of these are valid ways to get familiar with what's happening and what's coming. Brilliant. Well, thank you for sharing your deep thoughts with us on the show for a third time today, Vijay, and for giving us with resources to continue to follow along with after this episode recording. As always, really enjoyed the time with you and hope to have another session with you soon. Thank you, John. It was just an amazing conversation to have. Well, that was a tremendous episode. Always love when Vijoy Pandey comes on the show. In today's episode, he detailed how agentic AI has moved from isolated agents to connected agents, but connection isn't the same as cognition. Agents can exchange valid messages without

agreeing on what those messages mean. He talked about how every multi-agent session his team analyzed had at least one cognitive failure and that these failures compound so fast that three of them drop accuracy to just 16%, all while your observability dashboards stay green. He talked about how OutShift's open source observability and explainability platform, OXP, works like lane assist in your car, watching open telemetry traces for drift and triggering the Accord protocol to Lock-in Shared Intent, or the Concord Protocol, to resolve conflicts mid-task. He talked about how Cisco is scaling quantum computing horizontally by networking today's thousand qubit machines together, recently demonstrating entanglement swapping over 17.6 kilometers of Manhattan to Brooklyn fiber at rates 5,000 times better than anything achieved in a lab. And he talked about why running probabilistic AI software on deterministic hardware is a mismatch what we're doing today and how startups like Xtropic and unconventional AI are building chips that compute on distributions using thermodynamics and synchronized oscillators.

Very cool stuff. I hope V-Joy will be back on the show again soon. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for V-Joy's social media profiles, as well as my own at superdatascience.com slash 1041. That's today's episode number, yes, episode number 1041. Thanks to everyone on the Super Data Science Podcast team for making all of these wonderful episodes. Our team today is Podcast Manager Sonja Breivich, Media Editor Mario Pombo, Partnerships Manager Natalie Zajski, Researcher Serge Massis, and our founder Kirill Aramengo. Thanks to all of them for producing another outstanding episode for us today for enabling that super team to create this free podcast for you we are deeply grateful to our sponsors. You can support the show by clicking on our sponsors links, which are in the show notes. It really does make a difference. And if you'd ever like to sponsor an episode yourself, you can get the details on how by making your way to johnkrone.com slash podcasts.

Otherwise, help us out by sharing this episode with somebody who would enjoy the mind-boggling insights within it. Review the show on your favorite podcasting app or on YouTube. Subscribe. but most importantly, just keep on tuning in. I'm so grateful to have you listening and I hope I can continue to make episodes you'd love for years and years to come. Till next time, keep on rocketed out there and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon. Thank you.