Hybrid AI Infrastructure and the Rise of Non-Human Identity

We can identify an agent versus a human. That is a very solvable problem. It's the policy of like, oh, this agent can access JIRA, but it can only read code. Hi, everyone. I'm your host, Nick Lippis, and welcome to the Built for Trust podcast, where you get to hear from all the folks who are building and shaping AI enterprise infrastructure. Let's get right into it with our guests. Hi, everyone. I'm joined with Rob Kim and also with Tom Gillis.

We just finished a session on hybrid cloud, and so we want, not hybrid cloud, really, it's like hybrid AI infrastructure. Got to get the terminology right, you know? So we're going to like, we had a lot of unfinished business, so we wanted to actually talk a little bit more about it. We had Chris Moretti from Cigna on that panel as well, but he couldn't join us for the podcast. So anyway, Tom and Rob, say a quick hi.

Hi. Thanks for having me. Awesome. Great to have you, Rob. Rob, and also tell everybody who you're with. Rob Kim. So I'm with Presidio Network Solutions. We've been a long time Cisco partner and have now also expanded what we do into the data and application space, which obviously is quite consumed by AI.

Awesome. Tom, everybody knows you. Everybody knows you. Maybe on the podcast they don't. So I'm Tom Gillis. I'm the general manager for infrastructure and security at Cisco. So all our data center stuff plus all our network security, and we're a long time partner at Presidio.

Awesome. And Ono, actually. So it goes around, comes around. Awesome. Well, we kind of went through three phases that we were talking about. One was like the why for hybrid.

And so I think we kind of really hit that really, really well. Yeah. And then we started getting into the how, all right? And so what we were talking about is that Kubernetes is a really key core technology. And it's almost like it was, Rob mentioned this, and it made me think back to like college. You know, it's like you really don't understand algebra until you take calculus.

So it's almost like you really don't understand Kubernetes until you're into the AI world, you know, now. And so Kubernetes to kind of contain agents, Ethernet as the fabric, you know, connecting GPUs, storage, you know, together. And then also you started to talk about efficiencies, like a new round of products coming out into the marketplace that will give us, you know, 10 orders of magnitude or one to two orders of magnitude improvement. Yeah, one to two. Even a 5X is a big step forward. So I think the interesting state of the industry right now is that the bulk of the money that's being spent is going into these monster AI training facilities that are creating the frontier models.

Those guys are spending billions of dollars. And so they're burning, I don't know, 90% of the GPUs are going there. And that tends to skew the design center. So the current generation of silicon, it all has to be liquid cooled. Yeah. And so if you're, like, open AI or Google, you're like, no problem.

You know, like, they're building data centers next to giant, like, freshwater bodies so they can liquid cool these things. Yeah. If you're Cigna, you know, like, that's just not an option. Right? And so I think there's just natural waves to this, where as the enterprise starts to really understand the power of AI and how transformative it can be to their applications, there's going to be increasing demand for enterprise AI infrastructure. And it's going to be a derivative of what we see at the top of it.

It's kind of like Formula One racing. I don't know if you know this, but, like, the hybrid that you drive in your Prius, that technology came out of Formula One. Oh, wow. Yeah. It was called a kinetic curse, kinetic energy recovery system. And the first generation of those hybrids, it was mechanical.

So when a Formula One car comes into a turn, instead of wasting the energy by dissipating it as heat breaks, they built this little flywheel, and they put the flywheel in a vacuum that could go, like, supersonic. So the flywheel would go at, like, insanely high speed, storing incredible amounts of energy. And then they would, as it come out of the turn, they would release that energy back into the car and launch the car forward. Yeah. That's what a hybrid is. Yeah.

So instead of mechanical storage, a hybrid is electrical storage, converts it to chemical energy, right, in a battery. And then releases it back into the car. Same dynamic here. The stuff we're doing with these sort of monster training facilities, we'll make an enterprise version of it that will fit into the power and the cooling envelopes that are very difficult for enterprise to change. Yeah. For sure.

You can't just call it a power company. Fitting a data center. Yeah. I need another, you know, like, 100, you know, sort of megawatts. That could take years to get that. Yeah.

Absolutely. Yeah. So, Rob, why don't you talk a little bit about, like, you know, your deployment. And how this might change these building blocks. So I think the building blocks we're talking about is kind of a new efficiency model around compute utilization or GPU utilization, K8, Ethernet. I think those are the three major building blocks, right?

Yeah. Yeah. Well, I think the first thing is, is that we're no different than any other organization that started that journey in the cloud. Like, just to understand what can this pretty seminal technology actually do. And as we started to get more familiar, that's when we started to understand, okay, there are different ways to do this much more efficiently. The moment that was where we started to consider hybrid really being a possibility and feasible for us was when DeepSeq first came out.

When we started to see that you had a quantized distilled model that you can actually instantiate on a small number of GPUs, but still have the level of efficacy coming out of the model in terms of output, as well as similar speeds. And that was when we said, okay, there's a cause to do hybrid, especially if we can develop use cases and almost limit what we're asking for these models to do. Then with agent architectures and just standard service chain technologies or approaches, it was very simple for us to be able to then put these planned orchestrated workflows together in a way that it was just much more efficient to be able to deliver on-prem than in cloud. So for us, that turned our attention into, okay, well, how can we make sure that we're selecting an agent framework or platform that's going to be extensible? So allow us to be able to choose and utilize models during the build process where we may use something that's in the cloud, but during the scale and launch process, use something that might be much more quantized and distilled and something we deliver privately. Exactly. And so we just made the decision to go ahead and build our own. And in making that decision, containerization was definitely a key component of what we built out.

We then talked to our friends over at Cisco and said, hey, we have data center limitations, all the things that you talked about. We don't have time to put in liquid cooled and all of that, but we want to make sure that we get access to GPUs in a quick fashion, as well as be able to connect to all of our existing networking, not only from an access perspective, but also what we're building in that cluster fabric so that we can maximize, obviously, any sort of hardware investment we're making. Yeah. And you kind of entered this with an understanding that you were going to be distributing or offering multiple models to your users. So like how many models are you running right now? Well, right now, because it's relatively a new thing that we've launched. It took a while for as it is to get some of the hardware pieces.

And for us right now, we have Gemma, Huen, and Minimax that we've deployed. And Minimax is still a hefty model in its own, so it instantiates on four GPUs. And then we have plans to roll out additional distilled models as well. Yeah. Have you used Kimi? We have, yeah. So Kimi is another one where we're using actually as a part of a remote site deployment that we're doing.

And actually for software development, we're utilizing Kimi to potentially replace a lot of the things that we've done more based in things like CloudCoat. Yeah. Okay. All right. That sounds great. So, and then also, how are you kind of enabling a kind of switching or routing between the models? Is there like a kind of a router that does that? You know, so this really gets to like kind of tokenomics, right? So it's like those partitioning of tasks for high-level models, more expensive tokens to, you know, things that might be on-prem.

Yeah. Yeah. So the curated models are all API access. So, you know, that we actually contain the brokering of those services within our agent framework. So not in what we call our enterprise AI OS. So the things that, the on-prem models that are brokered through what we call Path. And so keeping that independent, it allows us to be able to take any agent that we're running, some personal assistant or whatever it might be, and just very easily switch out the model through, you know, drop-down selection pane.

So I can go from AWS Bedrock managed Sonnet to Path managed Minimax relatively easily. The big difference is, is obviously the network that gets served up is very different, right? One's more internet-level access layer, and the other one is going to be coming from on-prem resources that we're doing through a cluster fabric. Yeah. Okay. Great. Tom, like, you know, as you guys are kind of thinking through, like, kind of enabling, like, private AI stacks, you know, for enterprises, is that a place that Cisco will, like, participate in?

So you guys are, like, the best routing company on the planet, routing models? Yeah, yeah. Well, okay, so let's talk about that. What I think is really, really interesting is the way models are being used in applications, multi-models at one time. So right now, what most people do is it's a static switch. Yeah. I'm either using a low-cost, a premise-based model, or I'm using a frontier model, and you literally toggle back and forth between them.

But in very short order, I would argue, like, within 12 months, there's a thing called a model router, which is just a proxy. And today, it's just a proxy. So it's just pushing the traffic at one model or the other. But as we, the industry, whether it's Cisco or somebody else, other people working on this, put reasoning into that model router, it will then assign the, it'll choose the model based on the task that's being asked. So I often refer to software development.

In software development, if you're doing some complex design task, you want to point that to Mythos, which is the frontier model. It's capable of doing remarkable things there. It's uniquely suited for that task. If you're implementing dark mode, that's Kimi, right? And that can do it at a tiny fraction of the cost. And so tokenomics will drive that routing decision.

Security will drive that routing decision. Sovereignty will drive that routing decision. But it is a routing decision. Yeah. Right? For sure.

Yeah. But what's cool about this is if we think about, if you believe that that's going to happen, which it's not a leap to imagine that, right? Like perplexity is doing model counseling now. Yeah. So it's, you know, and I think it's just going to get better and better and smarter and smarter. And, you know, my view is like whoever sells a problem, good, because the app itself, it'll truly be multi-cloud.

Yeah. Right? That's like, because if you remember private versus public cloud 1.0, it wouldn't really have apps that span both public and private cloud. It was a little bit of that, but not much, right? Because it was like just latency and it was really what Gartner called bimodal IT. You had some stuff that ran out of the public cloud that was container and scale out and modern.

You had some stuff on the private cloud that was VM dominated. In the AI world, a modern app will probably live in both places. Yeah. Which I think is cool, but it opens up all kinds of interesting networking challenges. For sure. Right?

How do you segment that thing? How do you control that thing? You know, like, you know, where's the firewall go in that thing, right? And so, yeah, we're going to be busy. That is really, really interesting because you're right. Like, you know, the problem with, like, you know, doing multi-cloud, multi-cloud was really almost like the fun is that, okay, I have connections into various different clouds.

Yeah, exactly. And those workloads worked in those. Yes. And it really was dependent upon the developers. The developers would have familiarity and skill sets with the context and with the skill set or with the tool sets that were being offered by the cloud providers. This eliminates all of that.

Correct. Yeah. So, all of that gets eliminated. Correct. It's really, it's all, now with the interfaces, really around code development. Yeah, and NCP changed that if you add an A2A, the fact that you can have agent extensibility into any framework.

And that's really the key, right? Whereas before in cloud, if I'm developing things in native services that are just associated with AWS, it's very hard for me to be able to port that stuff over to Azure or Google or anybody else. Yeah. And not only from a technical point of view, but also from a cost point of view. Yeah, 100%. You know, and then the charging, you know, of moving data out of there, you know, as well.

Yeah. And so, I think we're talking about the whole data center architecture from the top down. Down. Right? And so, we're talking about the top, which is like, how do applications work? I believe that they're going to have dynamic, they're going to use multiple models and that the application of those models will be dynamic.

That's probably not that controversial, but it is generally not happening today. So, that's kind of like right around the corner. And that's super interesting. As we start to move down further into the stack, the hardware accelerators that we use, I believe there's going to be, you know, pretty significant divergence there. Right? And probably the best example of this is, you know, NVIDIA Bach Rock.

Yeah. There's a reason for that, is that inference looks different than model training. Yeah. And as you and I were talking about earlier, Nick, model training has been dominating the economics. That's the spend. Like, they're spending billions on training.

Inferencing is just kind of lighting up now. Yeah. But it's not at all unreasonable to think you're going to have TPUs and GPUs and, you know, plain old CPUs that are doing inferencing. And so, that too is going to look much more heterogeneous than the infrastructure that we have now, where, like, I buy this kind of rigid supercomputer and it's, you know, an app is inextricably linked to the box. Yeah. And this is, I don't think, I mean, you know, Rob, I don't know if you have a point of view on this, but, like, we're not talking about, like, five years out.

I'm talking about, like, months, five months out. Yeah. I'm thinking, like, the end of this year, like, into, like, you know, into the beginning of 2027. Yeah. Yeah. Yeah.

Yeah. Yeah. I mean, it's matching the workload to the appropriate, like, if you think about textual embeddings, right, it actually gets processed much better from a CPU perspective than being able to do that in looking at inference versus training, which one actually matches better. And so, that diversity becomes incorrectly. So, let's think about operationalizing. Yeah.

All right. So, like, we have traditional kind of operation centers, NOCs and, you know, and SOCs, and you have the folks who are kind of operationalizing or managing compute. How does that change in this world? So, we've been spending over the last two days a lot of time talking about agentic ops. Yeah. All these agents are going to be, you know, basically within containers.

Yeah. So, where does the operational piece kind of come into this? Yeah. Because there's, like, so many variables. So, now it's like there is clearly, it's a full stack. So, we have to think a bit of an operation model from a full stack and a multi-cloud kind of point of view.

Yes. So, I believe that so many of the operating principles that we, the industry, have held in a data center are going to change by force, right? Like, they are going to have to change, whether it's because of all the security vulnerabilities that Mythos has unleashed or because of this sort of new way of operating applications with this model routers and multi-cloud, you know, sort of deployment. And so, that is a pretty significant shift. And so, the other thing that is going to change is access control. So, these are both, you know, we're on the ONUG broadcast here, right?

So, let's talk about the networking aspects of both of those. In the data center, in the kind of Formula One sector of the market, like that big monster training facilities, these providers can use specialized network protocols for GPU to GPU communication. And they have a scale and scope and skill set to be able to do that. Yeah. As we start to move out of Formula One land and into, you know, Prius land, which is like for everybody. NASCAR?

Yeah, yeah, yeah, NASCAR in the middle. You know, we need to, I think that networking is a place that we, the industry, and the vendors can simplify. And this is what we've been working on with NVIDIA. Even though we compete, like they have their own, and a very thriving, you know, Spectrum X in the back. So, we're putting Cisco's software on Spectrum X, and we're showing that here at the show here, right? So, we want to make this easy for the enterprise.

And in my opinion, easy means Ethernet. Yeah. Right? Ethernet is the standard. It is one every single time there's any contention. So, hopefully, that's not that controversial.

But where we've been working with NVIDIA is that whether you're doing traditional data center networking, what we call front-end networking, stringing together storage and servers and, you know, firewalls and load balancers, stringing all that stuff together with a Nexus switch, or you're doing that back-end GPU-to-GPU communication, it's all a Nexus switch. We have programmability in our silicon that lets us operate in either mode, front-end or back-end mode, but it's the same CCIE, it's the same network management capabilities. You don't have to have some foreign alien network in order to run GPUs at scale.

Yeah. That's thing one. Yay. Now, on the kind of access side, as people start putting agents out there, that's a work in process. Yeah. And, you know, you and I have talked about this, Nick, a bunch.

When an agent wants to access something, we generally kind of push that towards human-based access control mechanisms. Yeah. So, a zero-trust solution. Well, if you think about zero-trust solutions for humans, it's pretty broad access. Like, at Cisco, when I log in, I can access hundreds, maybe a thousand applications, right, because of my role in the organization. And that access lasts for a long period of time, months, before I have to, you know, renew my password and credentials.

For an agent, I don't think agents can have that broad an access, because their agents have... The operation is fast. Yeah. For sure. Well, and they have good intentions, but they have no judgment, right? So, it has the broad access of a human, but the common sense of a toaster or printer.

And so, they can very easily make stupid mistakes. So, we, the industry, have got to put in place more dynamic, you know, use the word intent-based, which I like. You know, I think of it as more session or task-based controls, but you no longer get kind of carte blanche access to applications. And I'll argue, no one has really solved that problem yet. Like, we're working on it. For sure, yeah.

But, and I think we'll have a solution in months, but I think that's still an unsolved problem. I mean, to be clear, we can identify an agent versus a human. That is a very solvable problem. It's the policy of like, oh, this agent can access JIRA, but it can only read code. It can't actually check in code, right? Having that ability to apply what action the agent takes and monitor that, the only practical way to do that is with AI.

So, there needs to be a class of model that can be built and reasoning that can look at the interaction between an agent and the application or agent-to-agent. Or tool. Yeah, exactly. Agent tools. And say, this is appropriate, and this here looks like a deviation outside of normal.

Yeah. So, yeah. So, almost like kind of like runtime monitoring to like make sure the agent isn't going rogue. Yes. Or that those are the safeguards to prevent like data from being, or agent accessing data that shouldn't be accessing or maybe distributing, you know, content that it shouldn't be distributing.

Yes, very fine-grained controls. Yes. And so, kind of what's cool about that is that in this world, every access becomes a privileged access. Yeah. Right? We don't make a distinction between humans and, you know, printers and telephones and agents.

Like, every access is a privileged access. Yeah. And the network has got to be the thing that delivers that. It's very difficult to do that any other way. Yeah. Yeah.

So, it's in, so identity is in the network. Runtime monitoring is in the network as well. Yeah. Non-human identity. Non-human identity is in the network, you know, as well. Yeah.

It just logically lives in the network. You know, the guys at Pang will be like, oh, no, no, no, we do that too. And I'm like, okay, cool. Go do it. But human identity is going to live where it lives, which is generally in Microsoft stuff. Mm-hmm.

And that's not going to change. The non-human identity, I'll argue, the network is the thing that, it's the first thing that a non-human entity touches. Yeah. Right? An agent isn't going to raise its hand and log in.

Yeah, it's all headless. Yeah, it's all headless, right? It's going to connect to the network. And so, at the point it connects, the network has to be smart enough to be like, who are you? Yeah. What are you doing?

Oh, you're an agent? You're an agent working on Tom's behalf? Great. We establish that non-human identity. Mm-hmm. And non-human identity stores, there are wonderful mechanisms to federate that with human identity stores.

Yeah. Right? It's called an identity broker. That's a very mature mechanism where these things can talk back and forth. And it can look like one coherent view of identity. But we definitely need to treat non-human identities different than human identities.

That's interesting. So, like, really, what we're talking about is kind of a new stack. Yes. You know, kind of a new kind of hardware, you know, stack. A routing layer between models, you know, on top of that stack. And then, literally, an entirely new operational model for agentic overlays.

You know, that has to deal with, you know, agent identity, attestation. Yes. Across zones that you trust and across zones you don't trust. Yeah. Because this is a multi-cloud kind of environment. Yeah.

Also, run-time monitoring and kind of rogue detection of those agents. Yeah. Data guardrails around the kind of data that the agents can consume, ingress and egress. Yeah. A zero-trust fabric, you know, as well, where there's persistence, you know, for those agents, both in-zone and also, like, in-trust domain, out-of-trust domain.

Yeah. I would think there's got to be autonomy governance, you know, how much autonomy those agents can actually take as well. So, there's a whole new framework is really what you're saying. And also, what's the common thread in all that stuff? Yeah. It's the network.

Yeah. Right? Like, yes, the endpoint can play a really, really important role in that, but you can't rely on the endpoint. We know that, right? Like, you just can't guarantee that you have endpoint control. So, I'm not saying that the network supersedes endpoint controls.

Yeah. But the network is the one thing you can rely on. Yeah. That's the one thing you know you've got is the network. And so, having the network have an intelligence that it can tell the difference between a human and an agent and a printer, right? And then be able to dynamically apply a policy to that.

Yeah. That's what we're doing. That's what makes sense. It's like the network has always been the source of truth. Yeah. And of the entire IT supply stack or, you know, supply chain.

Yeah. You know, in an organization, it's like, you know, you can really look towards the network to understand, okay, is that really what's happening in the environment? Yeah. So, that makes, yeah, that makes sense. Yeah. Yeah.

All right, Rob, you buying this? Certainly, I mean, that's what we're seeing, too. In fact, when you start to look at how we're making agents extensible across gateways as well as even tool sets, you need that granular level control. I can't give you, you know, a full NCP tool service. I need to get down not only to the granular function, but then also the function at time, point of access. And all of those that created descriptors that become an ID all flow through the network.

So, it only makes sense that that's how we... Okay. The New York event is in the end of October. Yeah. Is that about the time we'll start to see, like, private AI stacks, you know, becoming available? Yeah.

I mean, there's private AI stacks today. Like, you guys are selling them, we're selling them. Well, yeah, true. But I think they will look very different in October. Yeah. You know, like, we're taking leaps forward in terms of making this stuff consumable for the enterprise.

That's really the... That's what I mean. That's the goal. And in that October timeframe, yes, I think there'll be a lot of significant developments that are not something you can touch today. That's how fast things are moving. Okay.

Yeah. Awesome. Yeah. Well, this has really been great. Yeah. Thanks for, like, actually extending the session into the podcast, you know.

So, this was, like, I think allowed us to do a little bit more of a deeper dive. Yeah. Well, you know, Nick, you and I were talking about this. There's so much change happening so quickly, and the network is so central to all of it that, you know, I think the Ono community needs to stay plugged into this stuff. Vendors, you know, partners, we all need to be communicating as much as we possibly can. Podcast is a great, easy way to do it.

Let's do more, right? Yeah. Yeah, for sure. Like, I'm actually starting to think, take that to, like, take that home. It's like, maybe we need a whole kind of, like, series of people doing podcasting, you know, just on this area. Because we've covered so many topics, each one of which is very, very significant in terms of, yeah.

how you run and build and operate a network infrastructure. Yeah, for the AI era. Yeah, cool, thank you. Tom, Rob, thank you very much. Everyone, thank you for plugging in as well. Take care.