Transforming an Old-World Organization

Enterprise IT is not known for its ability to be on the cutting edge. Traditionally, the technology stack in an IT shop can lag behind by five, or sometimes even ten years compared to what technology leaders are doing or thinking about. This trend isn’t totally unwarranted; the cutting edge got its name for a reason. However, there’s also a lot of bad or outdated reasoning for why an IT shop might use a “legacy” technology stack.

Automation encounters this hurdle all the time. There are often concerns from folks who want to automate but just can’t convince anyone else in their organization to take even the first steps. Or (and this is often worse because of the “bad reputation” it creates) someone started doing basic automation, it went wrong, and took down a revenue-generating system.

Unfortunately, there’s no easy answer to this. Each organization has its own battle scars and unique history. However, one potential answer can be found in a great book, How to Win Friends and Influence People. In it, Dale Carnegie hands down several nuggets of wisdom that are useful for life in general—but one theme that’s present in many places in the book is the need to see things from others’ perspective. To (heavily) paraphrase:

You can’t get anyone to do anything they don’t want to do. So you have to make them want to do it.

We’ll discuss this a bit more later, but in short, the business has to want the automation or in-house scripting. This cannot be some science project that upper management only finds out about when things go wrong.

Even if you start the right way, by communicating your automation strategy to the business, there will be opposition. Change always brings out the antibodies. It would be strange if it didn’t at least make someone feel uncomfortable. The important thing is to remember why you’re doing this—it’s not because automation is cool (even though it is), it’s for the tangible, measurable benefits it can provide to your business.

Another very important point is the need to do things slowly, building good, lasting engineering habits, and to set the right expectations from the beginning. Automation is not unlike healthy weight loss. Anyone that has been able to lose weight and keep it off will tell you it’s not about fad diets, it’s about fundamentals like eating the right things, with the right portions, and exercising. Short-term gains are not nearly as important as building healthy habits over the long term. Fad diets may show some short-term success, and they’re certainly flashy, but they aren’t meant to provide any lasting benefits or success.

Similarly, automation is incremental. If your organization is focused on putting together an “automation” or “DevOps” team, the effort is already doomed to failure. Automation is one of those things that needs buy-in across silos, and needs to grow organically over time. It is for this reason that those organizations that already heavily automate usually don’t really call it anything special. It’s just “modern operations.”

To recap, don’t try to boil the ocean or try to formally define everything you’ll need to automate in the next century. Just get started. Start small, and automate the simple stuff, even if it involves writing a few scripts and running them with cron, or focusing on automated troubleshooting initially. You’ll find there’s a lot more to do once you’ve simply gotten started, and you’ll have a lot more confidence to keep it up.

The Importance of Executive Buy-In

Again, it is very important that automation is done with a well-communicated purpose and strategy. That purpose has to focus on delivering value to the business—whether it’s better uptime, security, or just responding more quickly to changing business needs. Those metrics should already be tracked, and if you’re thinking about starting automation without these, you have the order wrong. The very first thing you should be addressing is how well you are communicating your short- and long-term technology goals with the business.

Once this is addressed, there are some very tangible benefits you’ll realize once starting down the automation path. First, any additional head count that’s needed for automation will be an easier pill to swallow. A very common complaint among engineers struggling to get automation started in their organization is the lack of resources to do it. With proper communication with the business, it will be widely understood that the cost of a bit of additional head count would pale in comparison to ongoing outages resulting from either totally manual process, or half-baked automation tooling that was written by an overworked engineer.

However, the single most important reason to have frequent, quality communication regarding automation with the business leadership is that when things go wrong—and they will—you won’t find yourself ripping out all those new tools and processes, but rather working forward to fix them. We’ll discuss this later, but one of the reasons the hyper-scale web companies like Facebook and Google talk about embracing failure is because they ensure they learn from their failures, and strive to ensure they don’t encounter the same problems twice. Each failure is an opportunity to grow. Getting the business on board with this plan from the get-go will make sure that any failures are not only learned from, but planned for.

As an illustrative example, Gitlab (the SaaS version of the software) famously had a significant outage in January 2017. Not only did the service go down, but restoring the service took 18 hours due to a series of previously undetected failures in their backup procedures. Rather than shut the doors and try to figure out what went wrong in private, Gitlab published a Google Document outlining everything they knew, and exposed it publicly so users could see what was going on. They even live-streamed their work to bring the service back online. Once the service was available again, they published an extremely thorough blog post outlining not only what went wrong and how they fixed it, but what they’re putting into place to ensure this event doesn’t happen again. This went a long way to assure customers that Gitlab was serious about their service and that they were interested in learning from mistakes.

The bottom line is that failures will happen and that automation doesn’t obviate the need for proper architectural design. Establishing a contract of transparency and frequent communication with the business leadership will help get the time and tools you need to build a proper foundation for an automation initiative.

Build Versus Buy

With all of IT becoming embroiled in the “open source movement,” it’s easy to get caught up in the hype. “Stop buying everything from vendors, and build everything yourself using open source software,” right? Unfortunately, this sentiment differs from reality.

Despite what the analysts might have you believe, big organizations like Facebook or Google—known for their automation chops—don’t build everything from scratch. At every part of their technology stack, they make compromises on what’s economical for them to “build” versus “buy off the shelf.” For instance, they may build their own servers for their huge data centers, but they don’t fabricate every single component from its base elements. They still have to buy something—they’ve just made the decision to go deeper into that stack. In contrast, they may find it perfectly acceptable to go with a canned solution from Cisco for their corporate wireless connectivity.

The point here is that everyone has to make this decision for themselves. It is likely that you are not at the scale of Facebook or Google, and therefore you won’t get the same benefits of building your own servers that they do, but it’s also very possible that you could benefit from taking on a bit more of the pie than you traditionally have.

A good rule of thumb to follow in Enterprise IT is that you can buy your way into 80% of the features you need. Vendors can’t cater to every Enterprise (try as they might), so they have to spread their engineering across their entire customer base and put out products that are good enough for the immediate use cases. This means that the technology stack you acquire in this way will leave about 20% of your specific use case unaddressed. Traditionally, Enterprise IT has just simply accepted this—but they don’t have to.

Even basic scripting can help fill in this remaining 20% feature gap. For instance—you may have a wireless controller that doesn’t generate reports the way you want. Instead of waiting for months/years for your vendor to change their UI (which may never happen), perhaps investigate if the controller comes with an API. Maybe you could write a Python script to retrieve this data and use a graphics library to generate some nice visuals for you. Note that this doesn’t mean you’re a software developer now—it’s simply knowing enough about scripting that you have an alternative to waiting years for your vendor to respond to your feature request.

Each side of the “Build versus Buy” paradigm tends to have its own traits (Figure 1-1). Organizations that choose to go with a commercial, off-the-shelf solution in one area tend to rely on external resources for support, whereas a technology stack that’s built in-house tends to keep the support model in-house as well, which relies much more heavily on promoting self-sufficient expertise in that area.

Figure 1-1. Build versus Buy

However, no technology decision is so binary. A good way of looking at the “Build versus Buy” paradigm—as with many other concepts in this lesson—is as a spectrum. No organization builds everything from scratch. Each technology team must figure out the right mix for their business. In reality, each organization will have a combination of these two strategies.

Learn What You Don’t Know

One of the most common reactions when we have conversations with peers or customers about what kind of things they can do to get started with automation is: “I didn’t even know that was possible!”

Indeed, working in IT can have a tendency to keep one in a bubble: constantly hearing different versions of the same message, and working on mostly the same things. This is one of the most dangerous scenarios for a technologist, because it’s worse than simply not knowing another technical discipline. In this scenario, you don’t know what you don’t know. It might never have occurred to you that you could use Python to talk to that switch in your broom closet, because those conversations just never made it into your world. If you stay in your bubble, you will have no idea what’s out there and will have difficulty growing as a technologist.

Fortunately, preventing this is easy. Frequently go outside your comfort zone. Jump into some kind of environment that operates outside your current “bubble.” There are an absolute multitude of technologies that may be interesting to you, and you may not ever even find out abut them until you challenge yourself to explore a new area.

This has tremendous value to you in your own career development, but it also essentially brings fresh ideas into your organization as well. This benefit isn’t always tangible, as evidenced by how difficult it can be to get approval to go to conferences, especially conferences that are outside your technical discipline. If you’re in a position where you’re in charge of deciding which conferences to invest in, realize that this is one of the least expensive ways to get fresh ideas into your own organization.

In short, go outside your comfort zone. Things don’t change that much in Enterprise IT, because our culture is very focused on and attached to IT vendors. These vendors have a vested interest in keeping things constant; rapid change doesn’t fit their sales model. One thing we can do to fight this is to stop getting all our ideas and guidance from vendors. Instead of going to your vendor’s big week-long marketing festival, maybe go to some smaller meet-ups, like your local network operator group. Or maybe some conferences outside your skill-set entirely, like a developer or automation conference.

Focus on Fundamentals

In any technical discipline, we always hear of new terms like “digital transformation” and “software defined” to describe the latest shiny new technology to enter the market. These terms give you a sense that you’re falling behind in the technology realm, and that buying the latest product (physical or virtual) will bring you back to the cutting edge.

In truth, most of us actually are a bit behind the times when it comes to technology. Especially in Enterprise IT, the technology stack can lag five, ten, or maybe even more years behind what’s considered the “cutting edge” stuff that folks in Silicon Valley are working with. However, buying the latest shiny product from your vendor never has and never will solve this problem for you. If this were the case, we would have solved this a long time ago. The real reason for stagnation in technology is an underinvestment in people and skill sets outside the traditional vendor-driven messaging we, at times, blindly follow.

Technology doesn’t really change that much. Speeds and feeds get bigger and better, and the industry can tend to go in strange directions at times, but it’s always a pendulum. Old patterns become new again, and the fundamental technology in use at the lowest level is usually the same. The TCP/IP you’ve known and loved from your earliest CCNA days is still very relevant in the latest Software Defined Networking products. The latest wireless products still reduce down to RF at their core.

This is one of the lessons that infrastructure professionals can learn from software developers. In general terms, infrastructure professionals don’t “build” as much as they “operate,” whereas software developers are accustomed to thinking like builders. To that end, software development is less of a skill than it is a collection of micro-skills. Just like a painter learns things like brush technique and the science of mixing colors, software developers pick up languages, tools, algorithms, and hardware knowledge, with the understanding that they will all become useful some day when the next big project calls for them.

These days, especially with the increasing importance of open source software in IT, the need for Systems or Computer Science fundamentals has never been higher. Learn about Linux. Explore a programming language. These fundamentals will help lead you to understanding more about what we’ve taken for granted in IT for so long. New IT products come out all the time, but they all run on hardware and software. So, whether you’re looking to make an entrance to a new discipline, or get deeper within your current one, focusing on the fundamentals is the best way to stay relevant across the vast, yet shallow changes in the IT stack over the long term.

Certifications?

Inevitably, we must answer the ever-popular question: “What is the value of IT certifications in the era of automation?” It’s an intriguing question, especially since it cannot be answered identically for everyone, as each one of us is at a different place in our IT career.

Certifications carry with them an implication that you know the material covered. So, while there are problems with IT certifications today, there’s no denying that there’s an interesting trade-off worth investigating. The certification will somewhat inflexibly define your capabilities in that area, but it’s something concrete and well-recognized by employers. Without certifications, you’d have to start every interview from scratch and prove to your potential employer that you know what you’re talking about. Certifications are a good way to short-circuit this, and certainly, if you’re new in IT, this is a very useful tool to have.

However, there are some limitations to what certifications bring to you. The value of this short circuit decreases over time as you gather more experience. In addition, certifications often serve the vendor first, so they won’t get you 100% coverage of everything you might want to know. You may find it useful to rely more on certifications at the beginning of your career, and as you gain experience, you can dive deeper into fundamentals, relying less on vendors to prove to employers what you know.

If you focus on the fundamentals, IT certifications become much more of a tactical tool than a career-defining education path. Certifications are perfectly fine for cutting through the initial stages of a hiring process, and for some employers, certifications are a requirement. However, understanding the fundamentals will not only help you win in an interview, it will also ensure that you continue to climb the technical ladder as the winds of IT change.

Won’t Automation Take My Job?!

One of the most common questions about automation is: “What will happen to my job?” Indeed, there’s a widespread belief that automation will mean a reduction in head count. After all, if a machine can do my job, who will pay me to do it manually?

This idea seems to be predicated on a few incorrect assumptions. The first of these is that automation is somehow an instantaneous, night/day difference, which is never the case. Automation is always incremental, and imperfect at every layer. You solve the easiest problems first, and gradually move up the stack to bigger problems. You occasionally go back and improve what you wrote last year.

Another incorrect assumption is that once automation is in place, there will be nothing left to do. This is also incorrect, not only because of the first reason given in the previous paragraph, but also because automation unlocks new capabilities you didn’t have before. It’s true that automation does eliminate the need for a warm-blooded human being to fill a certain role, but doing so creates new challenges that simply didn’t exist before, and those people should be reallocated to deal with the new problems. So while a certain role might be replaced by automation, there are always new opportunities opening up further up the stack.

So in a “post-automation” organization, it’s clear that roles and responsibilities will change. You still need good, well-trained people, they’ll just need to be reallocated to take on new challenges uncovered through the introduction of automation.