Speaker 1 00:00:01 Join us as we gather around the hedge, where we dig into technology, business, and culture with the finest minds in computer networking. Speaker 2 00:00:20 Well, hello Tom. I have to pause there for a second after the count in. It's, it's a thing now, so, yeah. Speaker 3 00:00:26 Yeah, because I was scrambling for the record button a few times ago. Speaker 2 00:00:30 . Speaker 3 00:00:31 Hi Ross. I'm doing great. How are you doing? Speaker 2 00:00:32 I'm fine, I'm fine. My editor also says there's not enough space between the one and the hello Tom, and sort of like, it's really hard to find that line, so I have to pause. It's, you know, it's an important thing, . Yep. So no new pictures, no new plants, nothing going on in your office. Everything looks exactly like it always does. That's, yeah, Speaker 3 00:00:55 That's, that's my life. Although I was thinking of getting a, getting a bookshelf to put back there and put all my Speaker 2 00:01:00 Books on. Oh, see, that would make a change. Then you could move the books around. Yeah. Yeah. Then, then that would make it different. Yeah. And today we are joined by Scott Robin Roben. I don't know how to say Rob Speaker 4 00:01:12 On Rob putting a robe on. Speaker 2 00:01:14 Okay. Awesome. Great. That's great. I mean, I'm terrible at this. So, and Scott is at the beach, which beach? Speaker 4 00:01:21 Uh, sand bridge, south of Virginia Beach on the beautiful east coast of the us. Speaker 2 00:01:26 All right, cool, cool. So north of the outer banks, it sounds like perhaps it's Speaker 4 00:01:33 The same land formation as the outer banks. There just happens to be this arbitrary, uh, state line between Speaker 2 00:01:39 Uh Oh, okay. Between Speaker 4 00:01:40 The top of it. That makes sense. And it's easier to get to than, uh, than the outer banks from where I live in Virginia, so. Speaker 2 00:01:46 Okay, cool. And we have Chris Gruman, right? Speaker 5 00:01:49 Yes. Grundman gunman. Well, I'm probably saying it wrong too. I mean, if you ask my German ancestors, they'd probably say it far, far differently than I do, so who knows. Speaker 2 00:01:57 Yeah, that's right. Yeah, that's okay. That's cool. And where are you, Chris? Speaker 5 00:02:01 Uh, I am in El Paso, Texas. I'm home this week. Uh, oh. I just got home, uh, I was with my son in Tokyo last week for his 21st birthday. Speaker 2 00:02:10 Oh, wow. That's really cool. Good. Yeah. That's great. And you have a whiteboard, man. I've been thinking about buying a whiteboard. I mean, it's like kind of crazy having a whiteboard. I should get a big one and stick it someplace. Speaker 5 00:02:21 I like it. I, I jump up there and, and I'm very visual though, when I think, so it's, it's helpful to me to kind of map stuff out and Yeah, Speaker 2 00:02:28 Yeah. No, it's good. It's good. And it's like set up in the camera, so you can actually bring it up close to the camera. Speaker 5 00:02:34 You could, yep. Speaker 2 00:02:35 . Speaker 5 00:02:36 I don't know if you could actually read it, uh, across the camera or not, but, uh, theoretically it's possible. That's, Speaker 2 00:02:41 That's, that's really cool. So, okay, today we're talking about network automation, which is a big topic, and so it's very hard to know actually where to start other than perhaps the importance of network automation. And, you know, for me, network automation's always about consistency and security and stuff like that. But maybe Chris or Scott have a different like, or a more involved detailed take than I might, Speaker 5 00:03:08 I mean, I can definitely jump in there a little bit. You know, I think, I think one thing is network automation is, is another, those things like maybe like the edge or, or even like cloud, that, that maybe means different things to different people. Um, and, and has evolved in its meaning over time. Um, my interaction with networks has almost always been mediated by some form of software and scripting. Um, so to me, like networking and automation have, have almost always just gone together, right? The, the first, uh, internet service writer I helped build was a wireless I s p, uh, up in Colorado in kind of underserved areas before D S L and Cable had penetrated out further away from Denver. And we built it on the cheap. And we actually built it by taking, you know, the, the core routers were actually Linux servers that were, you know, we were running a bunch of Pearl scripts to set up IP tables to set up Nat and, and static routes and things like that on boot. Speaker 5 00:03:58 So we kind of built our own routing oss. It was, it was very janky, and I wouldn't call it, you know, full routing os at all. Like, it was, was not on par with anything else. But, but, but, you know, it was kind of before Quagga and Zebra and all that stuff came out. So we just kind of do what we could. Um, and, and moving on through time, right? When I was working at, uh, TW Telecom, we, or Time Warn we're talking about at the time, I guess we, uh, you know, came in and, and, and wrote again, same thing, right? Pearl and Bash and Shells Scripts. We, we didn't really do any network maintenance without a script, right? We didn't actually go and change the configuration on, you know, couple hundred routers by hand. We would stage it all ahead of time with a script. Speaker 5 00:04:34 You know, it wasn't something that was a program by any major. It wasn't something that would be, um, used over and over again for the future or be maintained, but it was something we would write, you know, for that maintenance to, to use. Um, and in my opinion, I think when automation really kind of captured more people's attention outside of maybe the service writer community or, or a small group of folks that were working on really big networks was around, I guess it was what, 2012 when, um, OpenFlow kind of hit the scene and, and everybody started talking about S D N and at least in my opinion, people didn't care as much about control plane separation and data plane separation, and like the ability to be able to, you know, write new protocols on the fly in, in the network, which is a lot of what a OpenFlow promised. Speaker 5 00:05:15 They actually just wanted network automation, um, even, even though we started calling it S D N, right? And, and I think it's kind of snowballed since then. But, you know, that's one of the things that was pivotal for Scott and I is even since then, right? In the last 10, 11 years, network automation hasn't gone that much further. I mean, it's definitely progressed, but not in the way that I thought it would for sure. And I think collectively as, as a community, it's not, not really progressed the way we thought it would. I just, um, presented the results of a network automation, the state of network automation survey, um, at, at the last nano and the, the, the kind of headline there, not statistically relevant, but the headline of the, of the survey was about 40% of our networks are automated. Uh, and of course, I think a lot of people who haven't automated anything self-selected out and didn't answer the survey. So that may even be a rosier picture than, than it is. Um, so anyway, so I, you know, so, so as far as, you know, the importance of automation and all that we can talk about more about, but I think that's kind of at least my history of how we've gotten to here. Speaker 2 00:06:13 That's interesting. So you kind of worked on SSD Nish stuff before sdn and, and I would say that sdn, the other effort behind sdn, it's interesting that you bring it up because I'm talking about S D N A bit at work right now as well, is that SS d n we are forever trying to find a way to do traffic steering and traffic engineering in a way that's easier than what we're doing it today. Uh, and this is like a continuous quest in the networking world. How do I do traffic engineering? And I do it in a simple way. Well, it's not an easy problem, so there's not gonna be a simple solution. Sorry, I don't, I don't have one. But, you know, like even going back to M P L S and tag routing and all that other stuff, and back to a t m, all of those things are like traffic engineering. And I think you're right, to some degree, OpenFlow started as a way to automate the network, but automate it towards the specific end of doing traffic engineering. And, you know, we've still never gotten there really, Speaker 4 00:07:17 I don't think OpenFlow was a, was an interesting experiment, um, if you will. Like, you know, my bias, I was on the vendor side for a long time, you know, largely at Juniper, but sometime at Cisco Nokia and then another software startup. But, uh, you know, OpenFlow was way too primitive. Um, it was like kind of the assembly language, uh, analog, um, to a routing protocol. You could put very specific forwarding entries in and forwarding tables more appropriate for switching than routing, you know, maybe to overgeneralize, but, uh, you're not wrong. The quest for traffic engineering has, uh, Al has been with us from, uh, the days of a t m. Yeah. And there are a lot of folks that argue that, uh, you know, M P L S brought the complexity of a t m to the IP world. Um, yeah. You know, Speaker 2 00:08:07 It's, it's, yeah. Yeah. And I think, I think the failure of OpenFlow really was that, as you said, Chris, it's too low level. It's down in the act. It's like basically running P four today, you're actually programming the switch and nobody wants to be there from a network automation perspective, right? They wanna be at a higher level. Intent-based is almost, in my opinion, a little bit on the other end, right? It's kinda like, oh, I'm not, I'm just gonna tell you what I want the network to do. And I'm not, actually, it is kinda like, maybe there's a bit too much traction there. Maybe that on the automation side is a little bit too, um, I'm trading off convenience for effectiveness or, um, or for, uh, uh, for optimization a little bit too much in that realm, perhaps. Um, and actually, you know, my favorite was I two r s, but then I two o s went down a completely sideways orthogonal path to its original intent. And, you know, but having a standardized interface between the rib and the routing protocol would've been an absolute boon for network automation, I think. Because you could have said, I'm gonna let the dynamic protocol do what it wants to do, that's great, let's just let it do it thing, but I'm gonna override it here and there with certain things. Right? And, and that to me would've been a great thing. But we never . I was Speaker 4 00:09:24 Really excited about it for a while, for sure. Speaker 2 00:09:25 Yeah. Yeah. Speaker 3 00:09:26 I I like that. Speaker 4 00:09:27 I, Speaker 3 00:09:28 Go ahead. I, I like the, his kind of looking at it, looking back at the historical patterns, um, you can see the pendulum swinging. So you had, uh, you know, open flow and very, very, um, imperative, uh, touching of the network, um, and then swinging way back to a sort of intent-based. And I feel like the, the thing we should be shooting for is kind of as the pendulum crosses the middle, uh, grab, grab the middle. And if we can do that, then we can actually use, uh, the benefits of both sides of the pendulum. But I feel like when you're, when you're swung way to one side, you're, you're missing, and, and the benefits of the other side of the pendulum are kind of not available to you, just, you know, sort of in an intellectual sense. And I, you know, I think we, we sometimes we complain about the swinging of the pendulum. We're like, oh, we're back to mainframes. We were doing that 20 years ago, blah, blah. But no kind of old man talk . The Speaker 4 00:10:18 Cloud, the cloud is the new mainframe. Right. You know? Speaker 3 00:10:21 Right. But, but I think, I guess the point I'm trying to make is, as the pendulum swings, if we are, if we're paying attention, I think we can use that, um, momentum just intellectually to put us in the right place. I, I think I two r s is probably a pretty good example of, of the, the swinging of the pendulum and how can we figure out to apply the lessons learned to both sides. And I don't know, I think automation, I think network automation is, is well poised to be the center of the swinging of the pendulum. 'cause it's not extreme in either direction. I don't know what you guys think about that. Yeah. Speaker 2 00:10:49 If we can get the right interfaces, but Sorry, Scott, go ahead. Speaker 4 00:10:53 No, so, so to to Chris's point, you know, all the stuff that many of us have done in networking from our beginnings, you know, whenever it, whenever the beginnings were, uh, early nineties, late nineties, two thousands, it's all about automated mechanisms in the network, right? Is D H C P network automation. You bet it's right. You know, were you managing IP addresses on a spreadsheet early on? You were. And that didn't scale, right? Um, our routing protocols, automation absolutely. Right. And, and our link state protocols and improvement over rip. Yeah, sure. Right. So we've always been trying to find ways to make the network more resilient and free up fingers on keyboard time to do more interesting things. So that's, that's nothing new. But, uh, as Chris and I got to know each other earlier this year, like I, I heard he was doing the network automation survey that he referenced, and, uh, I said, that's interesting. Speaker 4 00:11:50 I need to, I need to get to know this guy. Um, and we had a couple conversations and, you know, we just kind of, you know, metaphorically looked at each other and said, why isn't the rest of automation in the network taking off faster than it's, and we basically decided, you know, we're both entrepreneurs and we can do something about it. We don't need to ask anybody's permission. Um, so that's why we formed this new, this new organization called the Network Automation Forum to ask the questions, what's the problem with the uptake? You know, what's the attenuation, what's the friction? And what can we do to facilitate it? Um, and to pour some kerosene on it and fan the flame a little bit. Speaker 2 00:12:32 Yeah. I think it's interesting because I know that other people have tried to work in this space. A lot of other people have tried to work in this place. The, the whole, the whole, um, what was it? Um, uh, not s r e, but network, network reliability engineer. Mm-hmm. effort at Juniper when I was at Juniper, um, was a big part of this as well, and trying to teach people to do stuff. Um, but actually, I wanna back up for a second. 'cause Chris, you said your first exposure to networking was through the automation side, which I find very interesting because of course, I come from the electronic side. I mean, I'm sitting here writing slides for a book I just finished, and I'm talking about modulation and how you modulate signals on ethernet and how you kill foreign crosstalk and nearing crosstalk and all that kind of stuff. Um, and so I, I find it very fascinating that you started on the automation side, kind of in an s D N world building a provider. So, I mean, that's just very interesting to me that that's where you came from. How do you see, I mean, do you see the rest of the networking, the world, like routing protocols and stuff is kinda like, come on guys, like really enough or you I mean, I'm just curious. I Speaker 5 00:13:48 Mean, I, I think calling what I was doing, uh, s d n is, is generous. Um, quite generous. Well, but Speaker 2 00:13:55 SD SDN means, but yes. Many things to many people, so, fair Speaker 5 00:13:58 Enough. But it was software driven networking, right? I mean, it, it was, um, and yeah, it's definitely been interesting to, to see that. And I mean one, you know, just as a kind of, you know, a side note, it definitely was a weird path in, I think, because one, it was, you know, I was building a wireless I s p and two, we built it kind of with, you know, like I said, with our own software and scripts and things. And so then I had to come back in and when I got into like a real I S P figuring out, you know, a t m and fiber and, and all the, all the stuff that people normally kind of learned first I had to learn after, uh, which was kind of wild. Uh, I kind of did like an upside down, uh, career move thing there. Speaker 5 00:14:33 But, but yeah, you know, definitely, you know, learning, networking, coming from that kind of, you know, building it on Linux servers and, and, and building it through scripting. One of the big things, and I'm not the first person to ask this question, but it definitely is front of mind for me, is why do the other kind of information technology disciplines have such a easier time with automation, right? And like, like db, you know, database admins, CIS admins, um, you know, just regular storage folks, just, you know, kind of, you know, across the board, almost every other practice outside of networking and maybe security, um, in kind of the IT space does a lot more automation typically than, than what we do in the network. And that's an interesting question, I think is, is, is looking at, you know, why that is. And, and I think there are some structural reasons for that. Um, but definitely that background kind of has led me to that question pretty early on. Um, and you can see, right? I mean, part of it is the hardware issue, right? It, it's, when you're working with a common X 86 platform, um, it's a lot easier to do automation than when every single device you're touching has a different chip in it, right? That that's, you know, o one piece of it or Speaker 2 00:15:41 A different operating system and right. And every operating system is custom built for a certain thing. And there is, I mean, you know, like Cisco iOS was brilliant when it was, was first written, but, and, and partially because we didn't have processors that could do things. I mean, honestly, let's be honest, the first processors we were working with in the, in, when we were dealing with Cisco iOS were, oh my goodness, Motorola, I have 80, 80 eights, right? Yeah. I mean, pre 80, 80 eights, I mean, like, so little processing power. It was crazy. And so, yeah, I mean, I kind of, you kind of get that, but Yeah, but go ahead. Sorry, I just, just occurred to me that Speaker 5 00:16:20 No, that was it. I mean, I think, I think, I think that's really interesting, right? I mean, I think, you know, you know, to your, to your point that, you know, my background definitely caused me to ask those questions, I think a little bit earlier on. Yeah. Um, and, and maybe see things a little bit differently. Um, which again, to, you know, to Scott's point, that's one of the things we want to kind of circle back around on and, and, and figure that out like today, are, are those constraints still in place, right? These historical things that cause networking to kind of go down a different path, a more manual path in a lot of ways, um, you know, are, are those constraints still there, or is it just in our minds now, right. Well, what's actually stopping broader network, um, deployment? I, I would say that, you know, I, I like Tom's analogy with, you know, with the pendulum and kind of looking at that and kind of how to ride that. Speaker 5 00:17:00 But I think one of the biggest challenges to automation is actually outside of automation. I mean, yes, the interfaces are needed, and maybe we still need better interfaces, but the lack of standardization as we were just starting to talk about a little bit, right? Whether it's at the processor and, and F P G A level or at the network OSS level, or at the levels above that, right? Everybody designs their network a little bit differently, right? They, they, they're, they're not, we're not stamping these things out of, of a factory at all. Uh, and then even once you design it, everybody configures it a little bit differently, right? Um, almost as a point of pride in some cases. In some cases, it's just, Hey, this is the way I learned how to do it, so I'm gonna do it that way, right? In other times, it's like, oh, look, I can use dot 55 for my gateway instead of.one. Well, cool. But now your network is less decipherable to everyone else who comes along after you, right? Speaker 2 00:17:46 Yeah. And, and part of it too is competitive pressure as well. I mean, you know, we tend to think that every delivery vehicle should look the same because it just carries boxes, right? That's all it does. It's just a delivery vehicle. And every car should look the same and every, you know, whatever, every ship should, it's really not that simple. Like you gain some competitive advantage, however slight it is by building something that's a little bit different. And that's part of the risk and reward part of business, right? If I build this network just a little bit different, I may actually gain a 2% competitive advantage, and 2% doesn't sound like a lot, but 2% compounded over five years or 10 years plus the other 2% that you do it, it makes a huge difference. And so, yeah. So we don't build, everything's all the same. Speaker 4 00:18:36 Uh, and there's a, there's one really clear use case, right? The financial services companies, right, who, you know, in trading situations know that latency makes a difference, and that goes all the way down to the optical layer of, of the networks they build, right? Yeah. Because that gives them a clear competitive advantage. It's not as clear financially in, in other verticals. But that's, that's a really good, you know, case for that. Yeah. Let me, let me insert here, you know, not to stop this conversation, but, uh, you know, we're here talking a lot about a lot of these topics. Um, and, you know, as you, you, you might be aware, right? You know, we're putting on a conference in November, um, November 13th and 14th, where we want lots of other input on these issues as well. You know, the four of us, um, are not the final word on all of this stuff. Speaker 4 00:19:26 And, uh, what Chris and I are trying to do, you know, in sponsoring our first event for the Network Automation Forum, auto Con Zero, um, I'll let you speculate on why we started numbering at zero. Um, you know, we're trying to get, um, folks together from across the industry, you know, customers, consumers of automation, solution providers and solution implementers, you know, like professional services firms that kind of bridge the gap and help, uh, help customers implement. Um, we want to get people together and talk about what's working and share, share stories, share information about how they've gotten around certain things. So sorry for the explicit plug, um, uh, so early on in the discussion here. But, uh, you know, we'll keep talking and hopefully it'll generate thoughts and ideas from folks that are listening. And, uh, if you want to hear more, you want to contribute more, you know, please join us in November. Speaker 5 00:20:19 Yeah. Yeah. And I think that, that kind of multi-stakeholder is, is really important, right? Because, you know, for whatever reason, and Russ, you bring up some good points that I hadn't thought of, right? That there, that this competitive advantage idea of, of, you know, having a little bit different network design, trying to push the envelope a little bit, maybe, which that's something I want to kind of put a pin in and come back to. 'cause I, I have a friend who has a really interesting theory on some of that. Um, but, but because of that, right? You know, we really need everybody at the table to move network automation forward, right? I mean, going back to Tom's point, right? We need these interfaces built at the OSS level, maybe even at the hardware level. I mean, um, you know, Russ, you made an interesting comparison earlier between OpenFlow and P four. Speaker 5 00:20:57 I think it's a good comparison because both of them were kind of rolled out and touted pretty widely. But really they were tools for vendors to build better products. I, I think, um, right? It wasn't something that operational folks really needed to pay attention to. Um, and it maybe tripped some people up 'cause they were like, oh, how do I use this in, you know, how do I, how do I, you know, use P four in my network? Well, you don't. Somebody's gotta do something with it first, I think. Um, and so the vendors have to be at the table, obviously, right? And then, you know, then there's these kind of overlay folks, right? And there, and there's a few vendors out there who are building kind of automation platforms, right? To go over the top and kind of maybe connect those different OSS together and give you this, you know, management layer. Speaker 5 00:21:32 Um, which I think looks something like intent-based networking. I don't know if those are the right words or not, but, you know, giving this kind of management plane across the heterogeneous network is really interesting. But then you really do need somebody in the middle tying those two things together. 'cause right now, because everything's designed and configured differently, even if you have, you know, if you know all the oss and all the chips at that kind of automation orchestration layer, you've gotta tie it together because you've gotta wade through those abstractions of, you know, which VLANs, you know, somebody decided to use, or whether they're using O S P F or I S I S or how they decided to configure it, right? And what they're waiting their timers at and, and all that stuff, right? Needs to be kind of, uh, figured out and, and and drawn together. So I think that's a big thing is, you know, we really need to all be talking together around the same table somewhere, um, to, to move this stuff forward. Yeah. Speaker 2 00:22:20 And, and unfortunately, I think the biggest blocker from my perspective really is just the a p i. We just don't have a common a p i today. We don't speak a common language. Yang tried, has tried for years, MIBs, et cetera, et cetera. P four was almost an attempt at the lower layers to do the same thing. Open floor was attempt at lower layers to do the same thing. But it's just, we just, like, for whatever reason, uh, it's performance or whatever it is, we always end up back to scraping the command line. And like, that just really sucks , it's really, yeah. Speaker 4 00:22:59 Right, right. Yeah. Well on the, on the, on the hardware access, right? Um, so, you know, that's a place where the big iron, um, vendors really try to provide competitive advantages over each other, right? Um, you know, Juniper has a history of their asics, Nokia has really, you know, developed a pretty consistent line through the FP series with FP four and now FP five coming out, you know, Cisco's reinvigorating it with Silicon one, and they're all different processors with different primitives. So I, I'm not holding my breath for any consistent a p i for actually programming forwarding asics and, and you know, Broadcom two with the Jericho two and Jericho three family coming out. Oh, and je Speaker 2 00:23:51 JE three ai, Speaker 4 00:23:53 Right? Well, right. Speaker 2 00:23:54 I mean, Speaker 4 00:23:54 Let's about that as an application area, right? Yeah, Speaker 2 00:23:56 Exactly. Speaker 4 00:23:58 Um, but, but you do need that hardware abstraction layer, right? Where that's where you're, I think you have the hope of getting consistency, um, across a different line of asics. The problem is gonna be because of that competitive differentiation. You know, you, you might get, you might have the lowest common denominator approach, you know, and I'll, you know, we could talk about, uh, netconf, right? Um, you want one, you know, one ring to rule them all. Um, that will only be consistent among the things that are common to all the supported platforms and that kind of vanilla ices things. Um, well and may Yeah, go ahead. Sorry. I Speaker 5 00:24:36 Mean, I didn't mean to interrupt, but that, that, that's kind of a, you know, the challenge we had with MIBs too, right? And, and it was solved with these kind of provider specific MIBs. Um, sure. And I, I, I don't know if there's a corollary there with net comp or not. Um, Speaker 2 00:24:48 Oh, no, there is, there is, there is. I mean, you have open config versus the I A T F standard config stuff, and that's just, you know, one provider said, we just don't, A big provider said, we don't really like the way the I T F is doing it. We're building our own. And now you have two. And now, like, as a maintainer for FFR routing, we're always running around going, okay, which one do we support? Okay, you know what we'll do, we'll try to munch the two together and make our own kind of, and now, so everybody has their own right. Which is what happens. And so, yeah. So Speaker 3 00:25:22 I, I think, I think because of this, so there's multiple, there's differentiation happening in multiple layers, uh, in, in the, in the market. So you have people who make products trying to differentiate between each other. And then you have people who consume products to build bigger, uh, higher order products, like very large providers or whatever. They're consuming the stuff. They're, so, they're consuming a highly differentiated landscape. Uh, and then they have to produce their own differentiation. So they're driving stuff down like you just described, Russ. So you had both of these things happening, like a common interface. I am pessimistic that it, that it will ever happen. That we have one thing where you can always get the, uh, you can always get the label table out of a router. Uh, with this, with this a p i i, I don't think it will ever happen. Speaker 3 00:26:04 And networking is just a very, is is a very, um, is a very, uh, specialized discipline forwarding packets is, is just not the same. Like, to kinda go back to Chris, what you're saying about why hasn't we caught on in networking, um, hosting a database can be done in a platform agnostic manner forwarding packets cannot ever, like we, we consume asics. And so because of that, I, I'm not trying to be like a downer, but, but I, I feel like because of that, the sooner we embrace the poss the, the fact that we have to provide the interface that we, the people building these networks have to provide the interface as soon as we accept that and stop complaining about interfaces, I think that's, that is one big thing that, that will get us to, um, yeah. To, you know, strong penetration of networking. I don't know this my theory. Speaker 2 00:26:48 Yeah, Speaker 4 00:26:49 It's a good point. Well, so there, let me, let me tail on that a little bit. You know, there's one horrible oversimplification that I kind of like to use for the history of network automation from, from Pearl to Python to platforms, right? And it's not just linear, right? But you know, Chris, Chris used the pearl word, right? 'cause that was, that was the scripting language for network automation. You know, back in the day, Python has gotten greater adoption, but there's this fear of lots of network engineers that I'm not a coder. I didn't take this job to be a coder yet. I have to learn Python now. Um, and there's, there is a level of resistance there that I think it's gonna take some fundamental change or, or may never be fully overcome. But Tom, to your point, that leaves room for other providers to come and provide automation platforms that have the, the magic word integrations, right? Speaker 4 00:27:45 Um, and I want, I don't wanna get too product focused here, but you know, in the conversations that Chris and I are having, you know, preparing for Auto Con zero, there's this family of companies that are rising up that have lots of integrations with the visibility vendors, the router vendors that make you not have to be a Python programmer to do automation at scale. Um, I think you're always gonna have some roll your own with Python, but the platforms, um, that are truly multi-vendor, they haven't been, um, acquired by, you know, some of the big network vendors. They, they're providing some really interesting, um, mortar between the bricks. That's my observation so far. I don't know what you think, Tom. Speaker 3 00:28:30 Yeah, I think mortar between the bricks. I like that. That's a Speaker 2 00:28:32 Great analogy. Yeah, I do like that. In fact, and, and that is another challenge beyond the single a p i, the second challenge, a challenge I hear a lot is I'm, I'm a network engineer. I'm not a coder. Right? Like, what, what, why am I learning code? Okay, well, I have a couple of answers for that. My first answer is always, um, learning code is good for you as a person. Eat the frog, , eat your vegetables. Okay? The frog, get over it. Speaker 4 00:29:04 Broccoli, yes. You put broccoli and not frogs. Yes. Speaker 2 00:29:07 Whatever. It's, whatever it is. I don't care. Might Speaker 3 00:29:10 Make some people squeamish, , Speaker 2 00:29:12 You know, Speaker 4 00:29:13 Just made me Speaker 2 00:29:14 , you know, come on. People like, really? I don't know. I know it's almost Speaker 5 00:29:22 Literacy at this point, right? It is. Speaker 2 00:29:24 I mean, yeah, exactly. That, that's the thing, right? Is like, it, it just, you just learn to code. I mean, really, even if you only learn Python, just learn it. And Speaker 5 00:29:33 The other thing is, you know, I mean, you know, a C L I is not that far from coding, right? Learning iOS, correct, yeah. Or learning Junos. Yes. I mean, it's, it's not that big of a bridge to Python, right? I mean, you already are kind of a coder if you're a C L I jock, if you're really in there pounding away every day Yep. You are writing code. Uh, I, I think, I mean, I don't know. Speaker 3 00:29:55 So I, I really like Chris that you used the word literacy, because there's a basic level of literacy. We all have to have to function in society, right? We gotta be able to read street science, we gotta be able to read the price tags at the grocery store, but we don't all have to be able to compo, you know, write a novel. We don't have to be able to read in another language other than our native language. Um, and I think the prob I think part of the problem is that what people see programming, they see it as this big huge discipline, and I have to learn all of it. And I have to, I've been doing this for 20 years, and now I gotta start over from zero with all these young kids coming outta college. I don't, I don't, I think people think that's what it is, and that's not what it is. Speaker 3 00:30:32 At least for me. What I've found is that if you, if you learn enough programming that you can speak their language, and then if you're fortunate enough to be paired with someone who's a really great developer, that that synthesis, like that synergy between you and them, I have built, I have built great things, uh, in, in a pair like that, um, that neither of us could have done on our own. Like some of the most satisfying work actually I've ever done in my career has been in that sort of arrangement. But, but if you're afraid of it, then you can never approach that relationship, I think. Right? Speaker 5 00:31:01 Well, and the other thing, I mean the, the, one of the secrets of why, I mean, I, I think anyway, why Python has taken off so well is the vast amount of libraries available. So, so what's wild is, you know, I mean, I, I think you're a hundred percent right, right? If you're sitting next to that person, or if you've got that person in your organization or, or, or somewhere where like, you've got this great developer available to you and you can work with them and you know, the language enough to kind of, you know, get the work done. That's amazing. I agree with you. Um, but, but you don't even have to have that, because that person, right, uh, has already written a bunch of code that's in a library that you can just call and you can write a really, really, really powerful Python script in like four lines, because you're calling like five other things that people have already built, you know, ahead of you. Um, which is pretty wild, right? And so I, I think, and that may be something that people don't quite understand, uh, if you, if you've never actually gone in and tried it, if you haven't actually, you know, done anything at all in Python, you may not realize that, um, you don't have to write it all yourself. A lot of it's already been written a lot of it. Speaker 2 00:31:56 Yeah. Yeah. Although, and I, I, I do think another piece of the coding puzzle that scares people is the process in the workflow. They don't understand. Like, I, okay, honestly, I still struggle with, with Git. I mean, it's so complex, , and there's so much stuff you can do and get, and like, I look at it sometimes and I'm like, I don't even know where to start really guys. I don't even know where to start. And so, and I've been doing coding for a long time. I mean, I've been, I started coding when I was in the, in the mid eighties, in the late eighties. I mean, that's, you know, writing c Speaker 3 00:32:29 I just wanna point out to the audience that they didn't already know. Russ is a, a maintainer of F R R saying this . So if you don't Speaker 2 00:32:35 , Speaker 3 00:32:36 If you're struggling with Git, you're not, you're in pretty good company Speaker 5 00:32:39 . Right? Right, Speaker 2 00:32:40 Right. Yeah, exactly. Speaker 5 00:32:42 You're not alone. And so, Speaker 4 00:32:44 I mean, Russ, if you hadn't gone there, I, I would have, right? So there's, you know, Chris rightly pointing out, you know, so much has already been done. And if you, if you are drawn to it and you have a, you have an ability, you know, to think like a programmer, be a programmer, you can do it. Um, I think another layer of the problem, um, is institutionalizing process and workflow, um, in a network operations organization. Um, there, i I, through this whole conversation, I see so many individual contributors who can go and get smart on Python and access all the tools that we've talked about out there and, and be super literate and get, um, and Terraform, et cetera. Um, but their organizations are relying on them as individuals to fix the network automation problem for them, where there are things that need to be institutionalized, you know, and where, where are the first level managers, the second level managers and directors that are fighting for making the process baked in, um, instead of just hiring somebody with Python skills is gonna solve my problem. Speaker 5 00:33:59 Hmm. Well, and that, that leads to another question, which I think is really interesting. Again, kind of, you know, to your point, Scott, the, there is this kind of business political, right? The layer eight, layer nine issues, um, that have a big impact here, right? So one is if you're in a, a large enterprise or even a, even a service writer, probably, uh, there's these different business units, and, and a lot of times they'll each kinda have their own networking team, right? If you're, if you're in a big enough organization, uh, and a lot of times each one is solving this problem differently. And so you'll actually, I I've come into some companies where I've seen, you know, maybe they're, maybe they're all using Python, and that's great, but they're definitely doing it in different ways, right? And maybe they're not even all using Python, right? Speaker 5 00:34:37 I mean, somebody's using Ansible over here and maybe they've got, you know, for some reason they were connected to a server team, so they're using Puppet over here, or they're doing things differently. Um, you know, so, you know, even just that consistency kind of across those things. Um, we've talked about this a little bit, uh, offline, Scott, this idea of a, a, uh, automation advocate, right? Um, where you've got maybe like this kind of traffic controller right in inside of an organization that's helping to spread the word, right? They're a little bit of a missionary and, and going to these, the different groups and saying, Hey, automation will, will help your life. Um, but also kind of spreading the, the standards and the processes across the organization as well. Um, so I, I don't know, Russ and Tom, if you've had any experience with, with a role like that, with, with somebody kind of coming in maybe for automation, maybe for something else, but having kind of that centralized, um, center of excellence type of, of, of approach, uh, to, to spreading these things and getting them right. Speaker 2 00:35:29 Yeah. But I think part of the problem is, is that we're engineers. I mean, I'm just being honest, , I mean, we don't, we don't, like, I wanna see Speaker 3 00:35:39 Where this Speaker 2 00:35:39 Goes. Yeah. We, we, we don't really like taking on those kinds of leadership cross, cross organizational, cross within the organization types of roles. It's just not in our personality type, unfortunately. And, and maybe this is something good that could come outta the conference as well, by the way, this, and, and the automation forum is getting people to be, to realize, again, you know, as Mike Buchan says, they're not soft skills, they're superpowers. Mm-hmm. And like, you know, don't be afraid. It's okay. You don't need permission to reach out to somebody in the other part of the company to go do something. Right? So I think, you know, one of the questions you asked early on, well, in the list of things we were gonna talk about was like, where are the leaders? Well, the leaders aren't there because we just don't naturally, we're engineers, we don't gravitate to that. We just don't. Speaker 3 00:36:32 There are some organizations that I think ha have done, I don't, I don't think this is the majority by, by any means, but there are some organizations that use the principal engineering role to do this sort of thing, um, where it's not architecture. It's not like sitting with and, and talking about, uh, the structure of the business or anything like that. But it's also not like, you know, operations and making networks run. It's, it's more of a technical leadership type of role. I, I, I developed one of these roles for myself and one company. I've had it in other, in other companies. I think that that is one way of accomplishing what you're talking about, Chris, is that their main job is not to make, make networks run and make packets move. It's to make sure that the correct ideas, um, get the, you know, get, get the right airtime in the organization and that the people, because there's lots of ideas out there, you really need the correct ones, including how to properly automate these systems. Speaker 3 00:37:22 Those are the ones that you want. 'cause people in general, I think are, are looking for, especially in large organizations where there's, so you had to get hundreds of people on board to do anything significant. Like it is, it is like walking through molasses. Yeah. It's, and so the, I think one big key to that is you gotta have someone whose job it is to del to deliver and sort of evangelize, uh, the correct ideas. And one of those should be like how to correctly structure network automation systems. Mm-hmm. I, I think that's one way to one way to do it. But, but that person, to your point, Russ, it's not just somebody who finds the most satisfaction in building things. They have to be someone who finds satisfaction in transplanting ideas. Um, well, like they Speaker 5 00:38:00 Would've been a technical marketing engineer, right? But you just focus them internally instead of externally. Yeah, Speaker 3 00:38:05 Sure. Speaker 2 00:38:05 Yeah. And, and part of the problem with that role is that I've discovered across time is that if somebody says, this needs to be done, and then you say, okay, I'll go do it because I'm the leader in that space and nobody else is gonna do it, you end up owning the code for the rest of your life, . And then you end up being mired in owning pieces of code that you've built across the years in order to solve specific, you know, know, like, okay, I can go build the connector between point A and point B. There's this great book called The Soul of a New Machine, which is all about Oh yeah, Speaker 4 00:38:42 Tracy Kidder. Speaker 2 00:38:43 Yeah, exactly. Yeah. And it's all about like, this one guy who sits there and builds interfaces. Everybody else is off doing their own thing. And that's great. And that's what you want to do. I mean, that's what you need in this space is somebody just to build interfaces and to put people together. But the danger is, is that, you know, you've gotta get your leadership on board to say, once I built the interface, somebody else has gotta own it. I, I can't own all the interfaces. Speaker 4 00:39:09 So to Tom's, you know, bravery and defining that role, , um, it, it, it's, it's, i i yay and amen to everything you said and the sustaining function. And it can't come down to just one person. That's, that's a, that's a death wish, right? Um, so I will gently challenge Tom and anybody else listening who has a similar idea or a different way of solving a problem, submit a proposal for a talk. Um, unfortunately when this comes out, we may have all our submissions in, but Tom, at least you right now, you're before the deadline. Um, I'm serious, you know, that this is the kind of thing that needs to be spread amongst the, the people that were, you know, will, will be coming to the event for sure. Speaker 5 00:39:55 Also because Russ's connation learn gi so that you can have other people work on your code . That's Speaker 2 00:40:00 Right. Speaker 4 00:40:01 That's right. It's a force multiplier. Rus, Speaker 2 00:40:05 Dude, Speaker 4 00:40:06 If I can learn it, I think we're about the same age , you know what? Fortune, fortune and sea programming in the mid to late eighties here. So, uh, um, I've, Speaker 3 00:40:15 I'm sorry, Russ, I did not mean to pull you into this Speaker 2 00:40:17 . I'm sure , Speaker 3 00:40:18 I'm sure Russ knows GI really well. Speaker 2 00:40:20 No act, actually, I really don't. And I need to go read a book on it because I just haven't spent the time. And part of that part of that's just time, and that's part of what I run into all the time. People ask me how I do some, so Speaker 4 00:40:30 Jump jump into, jump into the deep end of the pool. I, it sounds like you already have, right? , I mean, the only get skills I've learned have come from actually doing Right. Not, not reading about it. So Well, and, and Speaker 5 00:40:42 That raises another, oh, sorry. Go ahead, Scott. Speaker 4 00:40:44 No, I'm good. Speaker 5 00:40:45 Well, it just, it just raises another interesting point around automation and kind of the, the challenge of getting automation further deployed, I think, which is, there's a really good cartoon. I don't know where it originated, uh, I wish I could give credit to, to the cartoonist. Maybe somebody here can, but there's a picture of, of two guys really struggling to move this cart, and the cart has square wheels, so obviously they're struggling, right? Oh, yeah. And then there's a dude behind him that's holding up a round wheel and he is trying to get their attention and they wave him off and they say, no, no, man, like, leave us alone. We're really, really busy. We don't have time for your stuff right now. Um, and you know, to to to, you know, to your point, Russ, right? You know, there, it's, it's about the time. Speaker 5 00:41:23 And to your point, Scott, it's about what you're working on and it, it, it can, it can backfire on you though, right? Because I, I've definitely used it as you were talking about Scott, in that if I take on a project that involves something I wanted to learn, that's when I learn it, right? When I actually have to, when I, when I, you know, when, when I've gotta figure it out. But what that also means is I can get really down into the c l I and be cranking around in, in, in the guts of a router and, and not look up and say, oh, you know what? I could have solved this by spending, you know, half the time writing, um, a, a, a program or a script or something, right? And so, you know, Speaker 2 00:41:55 Yeah, it can cause blindness going back, going back to Tom's idea of a position. By the way, I learn a lot of stuff by teaching it. Mm-hmm. Sure. Because, because if I'm gonna stand in front of a class, I've gotta know four times as much as I'm putting on the slides, and I gotta be able to answer questions. So if you take on that part of that role as being, I'm gonna explain to people how this works. I'm gonna give classes in this. And by the way, you know, this is another permissionless thing, and I don't, I actually don't do enough of this, of, of just being permissionless about this, of just saying, I am setting up a video conference next Friday, and I'm gonna teach a class on whatever it is. I'm just gonna go do it, right? I do a lot of teaching outside of work, but I'm just saying, this is something you could do that would actually develop this leadership within the world in which you live to help, and it helps other people as well. So like, you know, give a class on get, I might come , . Speaker 3 00:42:54 I, I wanna strongly second what you're saying, Russ. I've done this in almost every job I've worked in, just because, I don't know, maybe I just like to hear myself talk or something, but like people, like, there's always somebody who, like, I, I've always had this thought that if I'll invest in this person, then they can take some of the stuff that I'm working on because they might be interested, but they just don't have the skills right now. And so I've always, every place I've wor almost every place I've set up the classes you're talking about Russ. And I've used different methods. Like my current one is a little, a little thing I do every week where it's just like I'm, I'm working on an open source nos. So the focus of it is how can we break this thing? How can we make it, bring it to its knees and make it malfunction? Speaker 3 00:43:27 Um, and that ends up being a super, super awesome way to teach. But then other times I've done it more formally, like, okay, we have this thing we have to deploy in the field, who wants to come learn about it? And there's always at least one. And sometimes if you have a class of you and, and two other people, like some of the best friends I've ever made at work were this kind of thing. Yeah, I bet is for professional development, it is way better than going to some class for a week or, or doing some webinar, like that sort of thing. I can really catapult. So yeah. Just want a second. Speaker 2 00:43:54 Don't be telling people not to come to my webinars, Tom . Speaker 3 00:43:58 Well, well, if there Russes go to Russ is just don't Speaker 2 00:44:00 In . Speaker 4 00:44:01 Think about, think about some of the terms we're kicking around here, literacy, education, permissionless, you know, these are all things that fall on management and leadership. Um, and they're not the same thing. So you've gotta use two different words. But, you know, create an environment where somebody's not gonna jump down your throat if you ask what you might think is a stupid question. And, uh, you know, shout out to Chris's, uh, podcast, the imposter syndrome network, right? Where, you know, everybody suffers from a little bit of, I don't belong here, and, uh, I never, a Speaker 2 00:44:37 Little is the wrong word. I'm sorry, Scott, a little is the wrong word. Yeah. Speaker 4 00:44:41 Right, right, right. Well, look, you know, my 20 years ago in the Juniper Tac, right? I mean, one of, one of the most educational experiences of my career, um, when you're an instructor, you get to teach how protocols are supposed to work, right? Um, but when you're in attack role, you see the worst of everything, how things break, how they actually work, . Exactly. Exactly. And I remember being scared to death to send, um, questions to support Dash private. Um, 'cause I knew there were escalation engineers that were gonna rip me a new one just for, oh, why didn't you R T F M, right? And, you know, you can make a difference as a manager and a leader by creating an environment where it's okay to ask questions, right? Oh, yeah. And it's, you don't have to ask permission to get three or four of you together to start learning get, et cetera, et cetera. Yeah. Speaker 2 00:45:34 Yeah. It's the same thing in Cisco Tech, by the way. You know, we had these mailing lists. Yeah. And if you, if you ask a question, you would get an answer. It would be, the answer would be between cuss words and, and, and other stuff. But you would get an answer. So you just had to put on your fire suit and say, I really don't know the answer. I'm just gonna like, it is what it is. Speaker 3 00:46:00 One, one of the things I noticed in, in these, in these type of environments, the classroom environments, is I, one of my methods I would use with people is I would say, okay, we need to make the network do this. So, so go ahead and do it. And, and it was something simple enough to do kind of on a whiteboard. And at first, when I would ask people to do this, they would be like, uh, and I knew that they knew how to do it, but they just didn't want to engage because this is the other side of that conditioning. The conditioning is I can't ask a question or I can't not know the answer, or it's not safe. Um, and then, but one thing I noticed is that I would, I would encourage them and, and, and finally get them to do it a couple times they'd do it wrong. Speaker 3 00:46:35 And I'd be like, okay, no problem. You're in the wrong direction, but let's go over here. After a couple of those, I would start asking those same people, okay, uh, do this task on the whiteboard or whatever, and they would make mistakes and they accelerated their mistakes, accelerated them. But the acceleration only happened. Um, just to, to come back to what you're saying, Scott, the acceleration only happened after people felt like, if I make a mistake, I'm not gonna get torched for it. Um, once, once they knew that it was safe for them to do, so making their mistakes made them smarter really fast. And that's, I think that's the thing that we have to, uh, create that sort of environment that's Speaker 5 00:47:10 Psychological safety, I think from the, the common literature. Anyway. Speaker 2 00:47:16 So Speaker 4 00:47:16 2013, Chris, Speaker 5 00:47:18 Very .