CULTURE – rule 11 reader: Recent Episodes
CULTURE – rule 11 reader
Network engineering is not “going away.” Network engineering is not less important than it was yesterday, last year, or even a decade ago.
But there still seems to be a gap somewhere. There are fewer folks interested than we need. We need more folks who want to work as full-time network engineers, and more folks with network engineering skills diffused within the larger IT community. More of both is the right answer if we’re going to continue building large-scale systems that work. The real lack of enthusiasm for learning network engineering is hurting all of IT, not just network engineering.
How do we bridge this gap? We’re engineers. We solve problems. This seems to be a problem we might be able to solve (unlike human nature). Let’s try to solve it.
As you might have guessed, I have some ideas. These are not the only ideas in the world—feel free to think up more!
If you walk into a robotics class, even an introductory robotics class, you see people … building robots. If you walk into a coding class, even an introductory one, you see people … writing software. If you walk into a network network engineering class you see … someone lecturing about the OSI model, packet formats, or how to configure BGP.
What problems are people learning to solve robotic engineering? How to build a robot and get it to do something to solve a real-world problem. What problems are people learning to code solving? How to tackle some real-world problem.
Sure, the problems being solved at an introductory level might be trivial, like: “Read this file and spit out a sum of the numbers in the fourth column.” But they are still starting, right from the beginning, by taking requirements and converting them into solutions.
What problems are network engineers learning how to solve? How to choose hardware, string it all together, and configure BGP.
Do you see the difference?
All engineers solve problems—it’s the nature of engineering. But are we creating a mindset in prospective network engineers, or even adjacent fields, that we solve real-world problems? Or are we giving them the impression that we solve whiteboard problems by talking about bits, bytes, configurations, and cable types?
Have you ever seen the glazing over of eyes while explaining how you put four transport protocols on top of one another (look at all the pretty tunnels)? How about when you create a chart showing how TCP and QUIC can be “kind-of sort-of” forced into the OSI model? Or when you spin out your BGP packet format charts, showing how we’ve (mis)used address families to carry everything anyone can imagine?
I’ve been teaching this stuff for years (okay, decades). Over time, I’ve moved away from teaching configurations and packet formats. I’ve gone from Advanced IP Network Design to Computer Networking Problems and Solutions. These are very different ways of looking at network engineering.
Focusing on real-world problems would help connect business and other IT folks to the network, connect theory to practice, and people to network engineering. Going home at the end of the day saying, “I solved a problem,” can be satisfying. Going home at the end of the day saying, “I configured BGP?”
Another thing adopting the mindset of solving real-world problems might do is help us lose unnecessary complexity. I know complexity is necessary to build resilient systems; we cannot build what we build without creating and encountering complexity.
But we often run ourselves into the ditch on both sides of the road.
We unintentionally build too complex because we try to make it too simple. Quick, which is simpler: building a data center fabric with one routing protocol or two? A single chassis system or several smaller fixed format devices? A proprietary system or something built on open standards?
How many balloons fit in a bag? (thanks, Don)
Failing to start with the tradeoffs, and thinking through what problem we’re actually trying to solve, leads to unnecessary complexity. Such designs might not immediately fail, but they will fail, and “it’s so complex” just isn’t an excuse.
Don’t even try to tell me there aren’t any tradeoffs. If you think there aren’t any tradeoffs, that just means you haven’t looked hard enough. Go find them, think about them, and document them.
We also build complex things because we think it offers job security, or it’s neat, or we like to feel like the kid who says to the world, “look what I built!”
I know it’s exciting to hear stories about that time someone rescued a network from a major failure—after all, that’s solving a real-world problem. Building a network that just works might be “boring,” but it solves many more real-world problems than raising a network from the dead.
We love our fashionable capes, but … capes can get caught in a nearby jet engine. Lose the cape. In the long run, it’ll make network engineering more attractive as a career field and field of knowledge.
The Bottom Line
No, the sky is not falling. We still need networks, and we still need network engineers.
Yes, there is a problem. Too many companies are going “to the cloud” because they cannot find people qualified to build and maintain their very complex networks. There’s too much centralization and too little oppeness.
So maybe let’s stop saying “we don’t need network engineers.” And maybe let’s really think about how we’re building things. And maybe let’s focus on solving real-world problems, starting from day one in network engineering classrooms.
Network engineering is still cool—let’s go out there believing—and selling—that idea to the world.
What does it mean to be a network engineer in today’s world of information technology? Phil Gervasi joins Tom and Russ to discuss the ins and outs of network engineering, and what it’s really like to be in this small corner of information technology today.
https://media.blubrry.com/hedge/content.blubrry.com/hedge/hedge-207.mp3
download
As we reach the end of what has been a hard two-year stretch for what seems like the entire world, Ethan Banks joins Tom, Eyvonne, and Russ to talk about the importance of taking care of yourself. In the midst of radical changes, you can apply self-discipline to make your little part of the world a better place by keeping yourself sane, fit, and well-rested.
https://media.blubrry.com/hedge/content.blubrry.com/hedge/hedge-206.mp3download
transcript
A long time ago, I supported a wind speed detection system consisting of an impeller, a small electric generator, a 12 gauge cable running a few miles, and a voltmeter. The entire thing was calibrated through a resistive bridge–attach an electric motor to the generator, run it at a series of fixed speed, and adjust the resistive bridge until the voltmeter, marked in knots of wind speed, read correctly.
The primary problem in this system was the several miles of 12 gauge cable. It was often damaged, requiring us to dig the cable up (shovel ready jobs!), strip the cable back, splice the correct pairs together, seal it all in a plastic container filled with goo, and bury it all again. There was one instance, however, when we could not get the wind speed system adjusted correctly, no matter how we tried to tune the resistive bridge. We pulled things apart and determined there must be a problem in one of the (many) splices in the several miles of cable.
At first, we ran a Time Domain Reflectometer (TDR) across the cable to see if we could find the problem. The TDR turned up a couple of hot spots, so we dug those points up … and found there were no splices there. Hmmm … So we called in a specialized cable team. They ran the same TDR tests, dug up the same places, and then did some further testing and found … the cable was innocent.
This set up an argument, running all the way to the base commander level, between our team and the cable team. Who’s fault was this mess? Our inability to measure the wind speed at one end of the runway was impacting flight operations, so this had to be fixed. But rather than fixing the problem, we were spending our time arguing about who’s fault the problem was, and who should fix it.
When I read this line in a recent CAIDA research paper–
“Measurement is political, and often adversarial.”
It rang very true. In Internet terms, speed, congestion, and even usage are often political and adversarial. Just like the wind speed system, two teams were measuring the same thing to prove the problem wasn’t their’s–rather than to figure out what the problem is and how to fix it.
In other words, our goal is too often Mean Time to Innocence (MTTI), rather than Mean Time to Repair (MTTR).
MTTI is not enough. We need to work with our application counterparts to find and fix problems, rather than against them. Measurement should not be adversarial, it should be cooperative.
We need to learn to fix the problem, not the blame.
This is a cultural issue, but it also impacts the way we do telemetry. For instance, in the case of the wind speed indicator, the problem was ultimately a connection that “worked,” but with high capacive reactance such that some kinds of signals were attenuated while others were not. None of us were testing the cable using the right kind of signal, so we all just sat around arguing about who’s problem it was rather than solving the problem.
When a user brings a problem to you, resist the urge to go prove yourself–or your system–innocent. Even if your system isn’t the problem, your system can provide information that can help solve the problem. Treat problems as opportunities to help rather than as opportunies to swish your superhero cape and prove your expertise.
Tom, Eyvonne, and Russ hang out at the hedge on this episode. The topics of discussion include our perception of security—does the way IT professionals treat security and privacy helpful for those who aren’t involved in the IT world? Do we discourage users from taking security seriously by making it so complex and hard to use? Our second topic is whether multicloud is being oversold for the average network operator.
https://media.blubrry.com/hedge/content.blubrry.com/hedge/hedge-153.mp3download
How do you balance loyalty to yourself and loyalty to the company you work for?
This might seem like an odd question, but it's an important component of work/life balance many of us just don't think about any longer because, as Pete Davis says in Dedicated, we live in a world of infinite browsing. We're afraid of sticking to one thing because it might reduce our future options. If we dedicate ourselves to something bigger than ourselves, then we might lose control of our direction. In particular, we should not dedicate ourselves to any single company, especially for too long.
Software Eats the World?
I’m told software is going to eat the world very soon now. Everything already is, or will be, software based. To some folks, this sounds completely wonderful, but—leaving aside the privacy issues—I still see an elephant in the room with this vision of the future.
Quality.
Let me give you some recent examples.
First, ceiling fans. Modern ceiling fans, in case you didn’t know, don’t rely on the wall switch and pull chains. Instead, they rely on remote controls. This is brilliant—you can dim the light, change the speed of the fan, etc., from a remote control. No unsightly chains hanging from the ceiling.
This last week I was talking to someone at a small startup that intends to eliminate all the complex routing from campus networks. In the past, when reading blog posts about Kubernetes, I've read about how it was designed to eliminate routing protocols because "routing protocols are so complex."
Color me skeptical.
Fear sells. Fear of missing out, fear of being an imposter, fear of crime, fear of injury, fear of sickness … we can all think of times when people we know (or worse, a people in the throes of madness of crowds) have made really bad decisions because they were afraid of something. Bruce Schneier has documented this a number of times. For instance: “it’s smart politics to exaggerate terrorist threats” and “fear makes people deferential, docile, and distrustful, and both politicians and marketers have learned to take advantage of this.”
I cannot count the number of times I’ve heard someone ask these two questions— * What are other people doing? * What is the best common practice?
While these questions have always bothered me, I could never really put my finger on why. I ran across a journal article recently that helped me understand a bit better. The root of the problem is this—what does best common mean, and how can following the best common produce a set of actions you can be confident will solve your problem?
Recent research into the text of RFCs versus the security of the protocols described came to this conclusion—
While not conclusive, this suggests that there may be some correlation between the level of ambiguity in RFCs and subsequent implementation security flaws.
Decision making, especially in large organizations, fails in many interesting ways. Understanding these failure modes can help us cope with seemingly difficult situations, and learn how to make decisions better. On this episode of the Hedge, Frederico Lucifredi, Ethan Banks, and Russ White discuss Frederico's thoughts on developing a taxonomy of indecision. You can find his presentation on this topic here.
Crossing from the domain of test pilots to the domain of network engineering might seem like a large leap indeed—but user interfaces and their tradeoffs are common across physical and virtual spaces. Brian Keys, Eyvonne Sharp, Tom Ammon, and Russ White as we start with user interfaces and move into a wider discussion around attitudes and beliefs in the network engineering world.
A while back, I was sitting in a meeting where the presenter described switching from a “traditional, hierarchical data center fabric” to a spine-and-leaf (while drawing CLOS, in all capital letters, on the whiteboard). He pointed out that the spine-and-leaf design is simpler because it only has two tiers rather than three.
There is so much wrong with this I almost winced in physical pain. Traditional hierarchical designs are not fabrics. Spine-and-leaf fabrics are not CLOS, but Clos, fabrics. Clos fabrics have three stages, not two—even if we draw them “folded” so you only see two apparent levels to the fabric. In fact, all spine-and-leaf fabrics always have an odd number of stages, and they are stages, not tiers.
BGP is widely used as an IGP in the underlay of modern DC fabrics. This series argues this is not the best long-term solution to the problem of routing in fabrics because BGP is not ideal for this use case. This post will consider the potential harm we are doing to the larger Internet by pressing BGP into a role it was not originally designed to fulfill—an underlay protocol or an IGP.
My last post described the kinds of configuration required to make BGP work on a DC fabric—it turns out that the configuration of each BGP speaker on the fabric is close to unique. It is possible to automate configuring each speaker—but it would be better if we could get closer to autonomic operation.
Every software developer has run into “god objects”—some data structure or database that every process must access no matter what it is doing. Creating god objects in software is considered an anti-pattern—something you should not do. Perhaps the most apt description of the god object I’ve seen recently is you ask for a banana, and you get the gorilla as well.
I tend to be a very private person; I rarely discuss my "real life" with anyone except a few close friends. I thought it appropriate, though, in this season—both the season of the year and this season in my life—to post something a little more personal. One thing people often remark about my personality is that I seem to be disturbed by very little in life. No matter what curve ball life might throw my way, I take the hit and turn it around, regain my sense of humor, and press forward into the fray more quickly than many expect.
Innovation has gained a sort-of mystical aura in our world. Move fast and break stuff. We recognize and lionize innovators in just about every way possible. The result is a general attitude of innovate or die—if you cannot innovate, then you will not progress in your career or life. Maybe it's time to take a step back and bust some of the innovation myths created by this near idolization of innovation.
You can't innovate where you are. Reality: innovation is not tied to a particular place and time. "But I work for an enterprise that only uses vendor gear... Maybe if I worked for a vendor, or was deeply involved in open source..." Innovation isn't just about building new products! You can innovate by designing a simpler network that meets business needs, or by working with your vendor on testing a potential new product. Ninety percent of innovation is just paying attention to problems, along with a sense of what is "too complex," or where things might be easier.
This week is very busy for me, so rather than writing a single long, post, I’m throwing together some things that have been sitting in my pile to write about for a long while. From Dalton Sweeny:
A physicist loses half the value of their physics knowledge in just four years whereas an English professor would take over 25 years to lose half the value of the knowledge they had at the beginning of their career. . . Software engineers with a traditional computer science background learn things that never expire with age: data structures, algorithms, compilers, distributed systems, etc. But most of us don’t work with these concepts directly. Abstractions and frameworks are built on top of these well studied ideas so we don’t have to get into the nitty-gritty details on the job (at least most of the time).
There is no enterprise, there is no service provider—there are problems, and there are solutions. I’m certain everyone reading this blog, or listening to my podcasts, or listening to a presentation I’ve given, or following along in some live training or book I’ve created, has heard me say this. I’m also certain almost everyone has heard the objections to my argument—that hyperscaler’s problems are not your problems, the technologies and solutions providers user are fundamentally different than what enterprises require.
Let me try to recap some of the arguments I’ve heard used against my assertion.
CHINOG is a regional network operators group that meets in Chicago once a year. For this episode of the Hedge, Jason Gooley joins us to talk about the origins of CHINOG, the challenges involved in running a small conference, some tips for those who would like to start a conference of this kind, and thoughts on the importance of community in the network engineering world.
One of my pet peeves about the network “engineering” world is this: we do too little engineering and too much administration. What brought this to mind this week is an article about Margaret Hamilton about the time she spent working on software development for the Apollo space program, and the lessons she learned about software development there. To wit—
Engineering—back in 1969 as well as here in 2020—carries a whole set of associated values with it, and one of the most important is the necessity of proofing for disaster before human usage. You don’t “fail fast” when building a bridge: You ensure the bridge works first.
Sounds simple in theory—but it is not in practice.
Let’s take, as an example, replacing some of the capacity in your data center designed on a rather traditional two-layer hierarchy, aggregation, and core.
In this episode of the Hedge, Tom Ammon and Russ White are joined by Ivan Pepelnjak of ipSpace.net to talk about being old, knowing about how things are going to break before they do, and being negative. Along the way, we discuss the IETF, open source, and many other aspects of the world of network engineering.
Network engineering and operations are both "mental work" that can largely be done remotely—but working remote is not only great in many ways, it is also often fraught with problems. In this episode of the Hedge, Roland Dobbins joins Tom and Russ to discuss the ins and outs of working remote, including some strategies we have found effective at removing many of the negative aspects.
For any field of study, there are some mental habits that will make you an expert over time. Whether you are an infrastructure architect, a network designer, or a network reliability engineer, what are the habits of mind those involved in the building and operation of networks follow that mark out expertise?
Experts involve the user
Experts don’t just listen to the user, they involve the user. This means taking the time to teach the developer or application owner how their applications interact with the network, showing them how their applications either simplify or complicate the network, and the impact of these decisions on the overall network.
Experts think about data
Rather than applications. What does the data look like? How does the business use the data? Where does the data need to be, when does it need to be there, how often does it need to go, and what is the cost of moving it? What might be in the data that can be harmful? How can I protect the data while at rest and in flight?
Nash King (@gammacapricorni) joins Russ White and Tom Ammon in a wide ranging discussion of ethics in IT, including being comfortable with standing up and saying "no" when asked to do something you consider unethical and the virtue ethic. This is meant to be the first of a series of episodes on this topic.
Over at the ECI blog, Jonathan Homa has a nice article about the importance of network planning--
In the classic movie, The Graduate (1967), the protagonist is advised on career choices, “In one word - plastics.” If you were asked by a young person today, graduating with an engineering or similar degree about a career choice in telecommunications, would you think of responding, “network planning”? Well, probably not.
Jonathan describes why this is so--traffic is constantly increasing, and the choice of tools we have to support the traffic loads of today and tomorrow can be classified in two ways: slim and none (as I remember a weather forecaster saying when I "wore a younger man's shoes"). The problem, however, is not just tools. The network is increasingly seen as a commodity, "pure bandwidth that should be replaceable like memory," made up of entirely interchangeable parts and pieces, primarily driven by the cost to move a bit across a given distance.
Once the shipping department drops the box off with that new switch, router, or “firewall,” what happens next? You rack it, cable it up, turn it on, and start configuring, right? There are access to controls to configure—SSH, keys, disabling standard accounts, disabling telnet—interface addresses to configure, routing adjacencies to configure, local policies to configure, and… After configuring all of this, you can adjust routing in the network to route around the new device, and then either canary the device “in production” (if you run your network the way it should be run), or find some prearranged maintenance time to bring the new device online and test things out. After all of this, you can leave the new device up and running in the network, and move on to the next task.
Until it breaks.
In a recent edition of ACM Queue, Kate Matsudaira has an article discussing the problem of being overly attached to a project or solution.
The longer you work on one system or application, the deeper the attachment. For years you have been investing in it—adding new features, updating functionality, fixing bugs and corner cases, polishing, and refactoring. If the product serves a need, you likely reap satisfaction for a job well done (and maybe you even received some raises or promotions as a result of your great work).
Attachment is a two-edged sword—without some form of attachment, it seems there is no way to have pride in your work. On the other hand, attachment leads to poorly designed solutions. For instance, we all know the hyper-certified person who knows every in and out of a particular vendor’s solution, and hence solves every problem in terms of that vendor’s products. Or the person who knows a particular network automation system and, as a result, solves every problem through automation.
Jordan, Eyvonne, and I sit down for a conversation that begins with meetings, and ends with talking about software defined everything (including meetings??). Outro Music: Danger Storm Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 3.0 License http://creativecommons.org/licenses/by/3.0/
It was quite difficult to prepare a tub full of bath water at many points in recent history (and it probably still is in some many parts of the world). First, there was the water itself—if you do not have plumbing, then the water must be manually transported, one bucket at a time, from a…
We love layers and abstraction. After all, building in layers and it’s corollary, abstraction, are the foundation of large-scale system design. The only way to build large-scale systems is to divide and conquer, which means building many different component parts with clear and defined interaction surfaces (most often expressed as APIs) and combining these many…
In my experience, simplicity is not valued enough in software development. Instead, there is a lot of emphasis placed on flexibility. —Felix Replace “software” with “network,” and think about it. How often do network engineers select the chassis-based system that promises to “never need to be replaced?” How often do we build networks like they…
If there is one question I get most often, it is “how do you get so much done?” One answer to this question is: I limit my use of social media. There is, another angle to social media use which is a bit more… philosophical. Some of you might know that I am currently working…
Mike Bushong and Denise Donohue join Eyvonne, Jordan, and I to discuss the gap between network engineering and “the business,” and give us some thoughts on bridging it. Outro Music: Danger Storm Kevin MacLeod (incompetech.com) Licensed under Creative Commons: By Attribution 3.0 License http://creativecommons.org/licenses/by/3.0/
Have you ever tried to make water flow in a specific direction? Maybe you have some particularly muddy spot in your yard, so you dig a small ditch and think, “the water will now flow from here to there, and the muddy spot won’t be so muddy the next time it rains.” Then it rains,…
We often use metaphors to describe a particular part of a thing or the thing itself. For instance, we might say “I’m as hungry as a horse,” to describe how much we think we could eat (although a more appropriate saying might be “as hungry as a bird,” as it turns out!). Network operators and…
I don’t often let my studies in philosophy and worldview creep into these pages intentionally. I don’t think it can be helped, of course, because the more I study philosophy, the more I see just how practical it is (contrary to popular belief). On the other hand, sometimes an observation about our world jumps out…
This is an interesting take on where we are in the data networking world— Tech is commoditizing, meaning that vendors in the space are losing feature differentiation. That happens for a number of reasons, the most obvious of which is that you run out of useful features. Other reasons include the difficulty in making less-obvious…
Networks are complex. But why? There are two fundamental reasons. The first is complexity is required to solve hard problems, specifically in the area of resilience. The second is that complexity sells. In this short take, I look at the second reason in a little more depth.
How many times, on reading my blog, a book, or watching some video of mine over these many years (the first article I remember writing that was publicly available, many years ago, was the EIGRP white paper on Cisco Online, somewhere in 1997), have you thought—here is an engineer who has it all together, who…
According to Roman philosophers, simplicity is the hallmark of truth. And yet, networks have become ever more complex over time. Why is this? Because complexity sells. In this short take, I talk about why complexity sells, and some of the mental habits you can use to overcome our natural tendency to prefer the complex.
Over at the Packet Pushers, Anthony Miloslavsky suggests that network architects have outlived their usefulness, so it is time to think of a new role. He describes a role called the “NRE” to replace the architect; the NRE would— …spend no less than 50% of their time focusing on automation, while spending the other 50%…
I have not counted the IETF’s I have attended; I only know the first RFC on which I’m listed as a co-author was published in 2000, so this must be close to 20 years of interacting with the IETF community. I’m pretty certain I’ve attended at least two meetings a year in some years, and…
In my latest short take over at the Network Collective, I explain the difference between engineering and metaengineering.
A very common mistake I see among engineers of all stripes is a failure to fully appreciate the value of time—both what it is worth, and how to make your time more valuable. What I normally see is something like this—I should be making $x/hour, because I have this specific experience, or that specific skill…