Dylan had to duck into a corner to get out of the way of the movers rolling the red tool chests out of Olivia's old office. He was able to inch a little further before the next group of movers hauled the red couch away as well. Once they were out of the way, he went into the large corner office to find Vic pacing along the windows of the office. The room was now full of plastic-wrapped furniture. Dylan was about to speak when Vic held up his hand and began talking. It took a moment before Dylan realized Vic had his wireless headphones in and was speaking with someone else.

The last several days had been a whirlwind. Olivia called an all-hands meeting just a couple days after the cybercriminal's last tweet, and she had announced that she was stepping down from her role as CEO. She thanked everyone for their support and announced that Victor Vega was going to be the interim CEO while the company conducted a national search. The most important thing, she emphasized, was that she didn't want any distractions from their new product launch. Even her.

“Tell them we're going to need double the number of orders to meet expectations. This will be our chance to do something revolutionary and we're not going to shoot ourselves in the foot by not having enough on day one to meet demand,” Vic said to the person on the phone.

Vic was the natural person to step up and assume the role. He had been at the company since the beginning and everyone assumed he'd take over the role as CEO when Olivia retired. And now the new CEO was calling Dylan in for a meeting.

“Make them believe, Cecilia. We're not selling Iron Man suits, but this is the next best thing,” Vic said, and tapped his phone to hang up and walked over to shake Dylan's hand. “Alone at last,” he said, tossing his blazer onto the top of the desk.

“Yes, we were giving Olivia regular reports on the status of Project Zero Trust. I can get you up to speed now if you want,” Dylan offered.

“That's not why I called you in. Sorry, there's no place to sit down yet,” Vic said, going to the window. Dylan followed. The view was spectacular with the leaves starting to change colors and fall.

“Security is costing us too much money, Dylan. This hacker thing is costing us millions to clean up right when we need to focus. Our investors are getting nervous. We planned to go IPO after the new product launched, and to make that happen we need people to believe we can turn this ship around. I'll be honest with you, everything is riding on the new product. If it doesn't blow people's minds, there's a good shot we go out of business. And we need to free up funds to make that happen.”

“Wait, does that mean you're killing Project Zero Trust?” Dylan asked.

“What? No. I expect results from your little project. I've read your reports, and I believe you're ready to finish the job. You've had a blank check so far and that has to change. I'm starting with the expensive consultant you've had training you for the past two months. Your team needs to vacate the briefing center, we need to start hosting meetings there again. And it's time you started working to secure our new product as well, that should be your focus right now.”

“We've got to show not just that the product is amazing,” Dylan said, “but that our next generation of customers can trust us enough to continue or even increase their subscriptions with us. Security is a part of the product. That's what we've failed to understand. I am to correct that. But we have to make the product secure with what we have.”

“You've got it exactly right, Dylan,” Vic said.

A few minutes later, Dylan and Aaron were walking together down a row of cubicles.

“Okay, but I don't know what the new product is,” Dylan said. Aaron had met him after the meeting with Vic and they were walking back to the briefing center. “People keep talking about it like it's a secret. And I didn't want to ask about it if I wasn't supposed to know. I figured someone would tell me eventually. But then, after a while, it seemed too embarrassing to ask.”

Aaron started laughing hysterically, crumpling over against the wall to catch his breath.

“Of course it's a secret, but this isn't Apple. Zero Trust isn't about not trusting people, it's about not trusting computers. But still, it's a good sign for you that MarchFit does have some good operational security.”

“Okay, so what is it?” Dylan asked as they walked back into the lobby on the way to the front of the building.

“Oh, no. You're not getting off that easy. You'll have to talk to your CTO.”

“We have a CTO? How am I just now finding that out?” Dylan exclaimed.

“He's one of those DevOps guys. They're like the Ricky Bobbies of the technology world,” Aaron said.

“What's that supposed to mean?”

“Ricky Bobby just wants to go fast.” Aaron chuckled as he said it. “Most security teams work for the CIO and are internally focused. CTOs are usually focused on the customers outside the organization. But the MarchFit app and the hardware need Zero Trust as well. DevOps is actually a great fit for Zero Trust because the process is already so well defined. This protect surface should be easy for you. You should grab Nigel and head over to meet Boris,” Aaron said as they reached the front doors. Dylan realized that this might be the last time they spoke.

“You knew? I thought you'd be more upset about leaving,” Dylan said.

“Two months is a lot of time to get up to speed on Zero Trust. You've got the right team around you. I figured this might be time for me to go spread the gospel of Zero Trust to other clients. But I'll tell you what. You can call me three times after I leave, no charge.”

A few minutes later, Dylan and Nigel were sitting in the waiting room outside Boris’ office. “You’ve got to be kidding me,” Dylan said.

“Our customers are all folks who are working from home. People aren't leaving their houses as much as they used to and VR is taking off. So having a 360-degree treadmill at their desk will still allow them to walk during business hours, but then they could watch TV or movies or play games while they run. Kids could go to virtual schools like in Ready Player One. You could run alongside your favorite basketball player during the game.”

“I thought you guys would be more upset about getting kicked out of the briefing center,” Dylan said apologetically.

“It was nice while it lasted. But we knew it wasn't going to last,” Nigel said as the door to the CTO's office opened.

Dylan followed Nigel into the darkened room. The only light in the room came from the cluster of computer monitors behind Boris and the blue lights from a fish tank that took up the entire wall on the other side of the room. There was a stick of incense burning behind Boris, giving the room a cinnamon and lavender smell. Dylan couldn't be sure, but it looked like the smoke detectors had been wrapped in aluminum foil.

“Did you know that 90 percent of the value of the S&P 500 is made up of intellectual property?” Boris asked. “That's what we're creating here. I trust all my developers to create value for the company. Nobody writes perfect code, but I think we should be measured on whether we can rise to meet a challenge. And because we have a DevOps shop, we were able to patch your hacker's zero-day vulnerabilities within forty-eight hours. At another company, we might have been struggling to patch six months from now.”

“That's incredible. Your team is amazing,” Dylan admitted.

“So what I'm most concerned about is the intellectual property theft here. What if our code was leaked to a competitor? What if our code was used by a criminal to hack our customers?”

“We're hoping to do something about that,” Dylan said.

“I've heard about that. But you know what I think? Zero Trust is just a fad, we can't operate without trust,” Boris said. “We rely on our people to solve problems every day. My developers won't work someplace where Big Brother is always watching them.”

“Zero Trust doesn't mean we shouldn't trust people,” Dylan said. “We're actually here to get to know more about you and your team. Tell me more about what makes your team successful.”

“We've adopted a DevOps model to deliver software to our treadmill network,” Boris said. “We're pushing hundreds of code updates every week.”

“That's incredible,” Dylan admitted.

“We can't do anything that will slow that down,” Boris said.

“Understood,” Dylan said. “We're here to help find ways to secure our code, but one of the first steps is to understand the process and how information flows through the organization.”

“Maybe start with the code repository?” Nigel suggested.

“Let me start at a high level and then work my way to that,” Boris said. “DevOps is about eliminating barriers between developers and operations. You should definitely read The Phoenix Project if you haven't already. It's like a religion for some. But the secret is that it relies on having the right tools in place to help teams rapidly and reliably deploy and innovate. There's a tech stack, and as Nigel indicated, it starts with the code repository.”

“That's where we keep the code?” Dylan asked.

“It does more than just store the code,” Boris said. “It handles version control and allows developers to collaborate on projects.”

“Where does it go from there?”

“The next step is the continuous integration and continuous delivery pipeline. When patches are sent, the code repository calls the CI/CD tool and this tool is what builds the application into a container. The container gets pushed out via our orchestration tool. And the whole thing is hosted in the cloud.”

“That's really helpful. How do your developers log in to all those applications?” Dylan asked. “We've been able to use our stronger identity protections to help secure processes from end to end. Are all of these DevOps tools using our identity system?”

“It takes too long to get accounts from operations,” Boris said. “We set up all the accounts ourselves.”

“Does that mean there are separate logins for each tool the developers have to use?” Dylan asked. Boris crossed his arms and nodded.

“Hey, boss,” Nigel began. “Wouldn't it be a great process improvement if our developers didn't have to log in to their accounts like fifty times a day? Dylan, for one code change, I might have to log in to all of these systems separately and they all use different passwords. I use a password manager, but some guys still type them from memory. I bet I spend twenty minutes a day just typing passwords.”

“Yes. Yes,” Boris answered. “This would help us go faster. I never trusted the identity system since we mixed customer and employee data. Thought it would be better to use local accounts. But now that we're separated, I'm open to this.”

“Great,” Dylan said. “One thing we promised Noor was that we would get rid of all local accounts everywhere.”

“If we integrate all our tools into SSO, it would also help us monitor for any suspicious behavior since we can better correlate user activity,” Nigel added.

“That's fine, but I don't want us to get lost in some philosophical discussion about how we need to write more secure code,” Boris said, turning back to his computer to continue typing an email.

“Okay. I can see that. I think Zero Trust is about getting rid of trust in digital systems. This means that we have to assume that we're already breached. Like what if the bad guys have already put some malicious code in our systems? Ideally, Zero Trust should help us with that.”

“Of course we want to write more secure code, but we're also on a deadline to launch the new treadmills,” Boris said as he turned back to look at them directly, crossing his arms again. “There's more hardware-specific code this time because the treadmills have to know what direction you're traveling, but also how fast. One improvement we're expecting later is for the texture of the treadmill to be able to change, like walking on sand or through snow. Or even being able to simulate slipping on ice.”

“Remember when we did that OWASP training last year?” Nigel asked. Boris nodded. “I've been thinking about how Zero Trust would really make a difference with those.”

“What's OWASP?” Dylan asked.

“OWASP is the Open Web Application Security Project that was started by Mark Curphey and Dennis Groves back in 2001,” Nigel said. “At the time, they realized that there were a huge number of vulnerabilities in websites, so they created a project that could give away free resources to developers to help them secure their code.”

“What does that have to do with Zero Trust?” Dylan asked.

“OWASP produces a regular list of the most commonly exploited vulnerabilities in web applications. These vulnerabilities all have one thing in common; they all exploit different aspects of trust in digital systems,” Nigel said.

“You're right,” Boris said.

“Take SQL injection, for example,” Nigel said, but seeing Dylan was starting to look lost, he elaborated. “SQL injection happens when an attacker is able to send commands to an interpreter because the developer didn't include proper authentication or input validation. The application should never trust input from a user directly. And queries should be parameterized using prepared statements.”

“Go on,” Boris said.

“Some applications have issues with broken authentication or broken access controls,” Nigel said, trying to catch his breath. “If applications are implemented incorrectly they could expose passwords, keys, or session tokens. Or they could expose other flaws that allow attackers to assume a user's identity. Or if permissions aren't properly enforced inside an application, attackers can exploit those to achieve unauthorized functionality. I've seen programmers just hide things from a user's menu rather than enforce security, and sometimes just viewing the source code on a web page is enough to get access to access other users' accounts, view sensitive files, change access permissions, or even modify other users' data.”

“Trust is a vulnerability,” Dylan said, echoing what Aaron said when they first met.

Nigel was talking fast now, while Dylan and Boris leaned forward to listen more closely. “Many times I've seen simple misconfigurations of security, or just not configuring security at all and using default configurations, including having open Amazon S3 buckets or using default vendor passwords. Or embedding passwords or other sensitive information inside the code itself. Or exposing sensitive information inside error messages.”

“Okay, okay,” Boris said. “I see how Zero Trust makes sense when it comes to the most common vulnerabilities. But we're also on a deadline here and we're not getting any additional staff to make our deadlines happen.”

“Do we still do automated testing of the code once it hits our CI tool?” Nigel asked.

“Of course, we have to automate testing. That's the cornerstone of DevOps,” Boris confirmed.

“What if we put some automated security testing into the CI tool? We could do some light OWASP scanning to make sure that we weren't introducing any delays.”

“I don't have any objections to this,” Boris said. “Maybe we could do some authentication testing as well. Make sure we're searching for hard-coded data like IP addresses. We'll be introducing some additional enhancements when we release Ben Richards later this month. We should be able to work it in by then,” Boris agreed.

“Ben Richards? Isn't that the name of the character Arnold Schwarzenegger played in the movie The Running Man?” Dylan asked. “I love that movie.”

“Of course. We name all our major builds after famous runners. The last version was codenamed Usain Bolt. The one after Richards is going to be called Florence Griffith-Joyner.”

“Flo-Jo? Respect,” Dylan said.

“Did we just become best friends?” Boris asked.

“And now you're quoting Step Brothers? What is even happening right now?” Dylan said, chuckling.

“Nigel, you didn't tell me Zero Trust was cool. Tell me what else you need to know.”

“What about secrets?”

“I've heard that millions of API secret keys are leaked every year because developers wrote them directly into their code. Cybercriminals now scan public repositories and collect those API keys. So we need a way of storing and managing those secret keys that doesn't involve sharing them via email.”

“You seem like you might already have something in mind.” Boris grinned.

“We should use a scanning tool to detect any API keys or other hard-coded data that's stored in our repositories,” Nigel said. “We can use a secret manager to help manage our keys. Rather than hard-coding secrets in our docker ENV files, then sharing them over Slack or Teams, the secret manager will manage those permissions for us.”

“Dylan,” Boris began, “I'm surprised you haven't asked about our Kubernetes container orchestration system yet.”

“What about Kubernetes?” Dylan asked.

“I'm glad you asked. Kubernetes isn't secure at all by default. So we've done a lot already to make sure it's secure. Just like we used to do back when things were all in our data center. We use network segmentation to separate clusters from other workloads and to separate workloads from the Kubernetes control plane. This is only possible if control and data plane traffic are isolated from each other. There should be a firewall between the data and control planes.

“I've also heard that role-based access control isn't enabled by default in Kubernetes. But we can do that now that we're going to integrate with our identity service, right?”

“Yes, this is a lot of work,” Boris admitted. “But I actually feel much better about the new product launching now. Not to add more to our plate, but we purchased a tool last year to do runtime security, but we haven't deployed it yet. Is that something you guys can help with?”

“Of course. I'll talk to Isabelle. She'll want some information about the project—can you tell me more about what it does?” Dylan said.

“Since our application runs in the cloud,” Boris said, “we've found we actually have less visibility into what the app is doing. I'm worried about container images themselves being compromised before we get them.” “But also we need to have a way of figuring out whether our runtime environment is secure so our containers aren't running in privileged mode or getting access to data that a hacker could leverage in a breach. So if our SOC is working the incident response plan, they need to have detailed audit trails of all commands, what files were accessed, and what sessions were in use at what time.”

“That's great since we already have it, there won't be any added costs. I think Vic will appreciate that,” Dylan said. “We will be meeting with our SOC team soon, but I'll include them in the project.”

“What if we can't get to a particular vulnerability for a few weeks? Is there anything that we can do in the meantime to protect our application?” Boris asked.

“At my last job,” Dylan said, “we had just completed rolling out a web application firewall or WAF to protect all of our websites. Most WAFs can protect against the most common OWASP attacks or even help detect and prevent credential stuffing attacks where cybercriminals use exposed usernames and passwords from other breaches to break into other services that customers might use.”

“We've looked at WAFs before,” Nigel said. “The problem was always that they were a proxy. They introduce another single point of failure if they aren't already integrated into our load balancers. And if they aren't integrated with our existing load balancers, then you have to have some manual process to sync certificates around. And the ones that are integrated with our on-premise load balancers won't work with our cloud apps.”

“That's true,” Dylan said. “But the one we implemented could use an agent on each endpoint. But really, WAFs are just a Band-Aid. We should be including web application testing in our penetration testing schedule to augment our regular testing.”

“Dylan, you're really making me think today,” Boris said. “One of the pillars of DevOps is the idea of infrastructure as code,” Boris said. “We can code instructions on how we expect the environment to work, and then it works without taking weeks or months for a sysadmin to get around to configuring a VM. What if we started thinking of security policies as code as well?”

“Boss, I think you're on to something,” Nigel said. “Defining security policies in our code will help us scale up as we roll out features into our different testing and development environments before they go to production. And the security policy code will get checked into our code repositories so it can be versioned, and we can easily recover if something goes wrong.”

“What kind of security policies are you thinking about?” Dylan asked.

Nigel was on the edge of his chair now, talking rapidly. “We can define roles and permissions inside code. Enforcing least privilege and separation of duties in code will help keep our developers focused on writing code rather than explaining unique requirements to sysadmins. And it can be audited much more easily since the configurations will be centrally stored rather than being spread across a bunch of different systems.”

“This could also help us make sure our developers are never able to log in as root,” Boris added. “We've had issues enforcing this in the past. Developers should also be able to assume different roles for testing. Automating this process could really help speed things up.”

“One of the things that we are looking for is opportunities to seamlessly inject reauthentications into the process,” Dylan said. “Could we also require a new MFA when someone needs to do something major, like pushing new code out?”

“Yes. We can enforce a lot of things in our security policies, like time-of-day access, restricting IP addresses, or requiring SSL connections,” Nigel said.

“The final thing that we need to make sure we've got covered is monitoring,” Dylan said. “From talking to Noor, we've got our logging pipeline set up for the whole development environment from the code repository to our cloud infrastructure. And when we shift to using identity, this should make it easier for the SOC to correlate events.”

“So what else do we need to work on?” Boris asked.

“When is the last time we did a code review?” Dylan asked. “Ideally we should be doing both static and dynamic analysis on a regular schedule like we do for our other penetration testing.”

“It's been a couple years,” Boris admitted. “But you're right, this is something we should do soon. We don't have a tool that will do this; is that something your penetration testing vendor can help us with?”

“I assume so, but I'll check with them and let you know,” Dylan said.

“The last time we did a code review, it didn't really discover anything that we didn't already know about. I honestly couldn't tell if the tool didn't work right or whether the guy running the tool didn't know what he was doing.”

“I think we should be taking a belt-and-suspenders approach to testing. I think it's okay to have a couple options on keeping our pants up. Have you ever considered using a bug bounty program?” Dylan asked.

“We proposed this last year,” Boris said. “It would take at least three, maybe four developers to manage the program. Half a million to start even before you get to the bounties wasn't going to fly. It got turned down.”

“What about the managed bug bounty program companies? They can manage it for us and aren't nearly as expensive,” Dylan said.

“We would still need to have someone manage the company and respond to any issues that come in,” Boris said, turning to Nigel.

“I'm in. Let's do it!” Nigel said.

“That will help us make the business case. So all our developers have to work on is fixing issues. We can even work with April to make a press release. This could help change the narrative around how MarchFit is responding. I think Vic can get behind that. Making a real difference for security at a fraction of what we thought it would cost in the past.”

“How do you want us to engage your team if the pen testers or a bug bounty hunter comes up with an issue?” Dylan asked.

“Well, it has to integrate into our existing process. I don't want our developers to have to log in to the help desk ticket system to look at a bug tracking report,” Boris said.

“Boss, I'll be the embedded security liaison into the Zero Trust team,” Nigel said. “We'll only engage the team using the same Jira tickets that the team already uses to track other issues.”

There was a knock at the door, and Harmony stuck her head into the room. “Hey, boys. Sorry, but I need to talk to Dylan for a second,” she said. “Why's it so dark in there? I'm not interrupting anything, am I?”

“Boris is cool,” Dylan said. “We don't have any secrets here. You can just tell us.”

“Oh, okay,” she said, opening the door all the way to let the hallway light in. “We just got a call from our SOC. They're seeing some weird anomalies. Like we're under attack or something. I thought I'd bring you in before I talked to them.”

“Is it Encore again?”

“We don't have any way of telling that for sure,” she said.

“What, no tweets this time?” Boris joked.

“Nothing on his account. I was hoping it was just scanning, but the logs they sent over look a lot more sophisticated than what we've been seeing from him in the past,” she said.

Key Takeaways

Application security is a huge challenge for many organizations. DevOps has revolutionized rapid delivery and continuous improvement of software for organizations; therefore, DevOps can also be an effective partner in delivering Zero Trust. Building Zero Trust into the process itself ensures that security isn't adding additional delays into the application development process.

In this case, the protect surface is the entire DevOps environment, and the transaction flows are all related to the development and deployment process. Often, applications are deployed with service accounts or by developers directly. When identity and access management aren't integrated with the protect surface, each developer may have reused personal usernames or passwords; passwords may not meet complexity requirements; and two-factor authentication may not be enforced. Single Sign On can be a way of reducing the number of logins for users, increasing productivity and improving security.

In some custom applications, developers want to embed identity in the application itself. This should be avoided for a number of reasons. Passwords shouldn't be stored inside your own database. If your app does perform its own authentication, it needs to include a lot of additional custom features, such as password reset or multifactor authentication, that can be better provided by a dedicated identity service. And using a dedicated identity service also ensures that the authenticator for your app stays up to date and secure since in-house resources might get pulled away to maintain other parts of the application.

A big part of integrating Zero Trust into a DevOps environment is integrating with the existing culture of the organization. In the example in this chapter, Nigel is the embedded security team member inside development, but he has been a developer for a number of years and is passionate about bringing security and developers together. The security team will use Jira tickets that the developers already use to be a part of the process. And security testing can be integrated into the testing process as well as at regular intervals.

Traditionally, many of the most common vulnerabilities in the OWASP Top 10 vulnerabilities have been directly attributable to exploiting trust in digital systems. Each year, the OWASP foundation updates the list with the most current attack vectors being seen in the wild.

Although Zero Trust doesn't explicitly address techniques to write more secure code, removing the trust relationships around the development process can go a long way toward improving security. Even just removing secrets from being stored in the code directly—like IP addresses, API keys, or passwords—can go a long way toward containing any intrusions. And by integrating with identity services, developer permissions can be limited to only those necessary, further limiting the blast radius of any intrusions.

DevOps is also very focused on using the cloud to rapidly deliver services. Orchestration tools like Kubernetes or Docker can help facilitate scalability, but they also introduce new security challenges that traditional security teams or infrastructure engineers aren't aware of. But there are numerous hardening guides available for these tools, and once you do apply the appropriate security controls, those controls can scale as well.

We don't expect all code to be secure when it enters production. This is where a web application firewall (WAF) can help. WAFs are more aware of applications than traditional firewalls and can do more to protect websites as well. Modern WAFs are able to learn how applications work and can help enforce input validation, detect malicious activity like SQL injection or cross-site scripting or “XSS,” and block that activity before it is exploited. But WAFs should only be used as a Band-Aid, so when a vulnerable page or application is found, the underlying vulnerabilities should be fixed as quickly as possible.

DevOps assumes that any issues with code can be rapidly addressed. After a former NSA hacker publicly announced a zero-day vulnerability in Zoom at the beginning of the pandemic in 2020, the company shifted and was able to issue a patch within 24 hours in part because Zoom uses a DevOps development model. DevOps can help improve security rapidly, but the organization needs to be looking for security flaws continuously. Static and dynamic code analysis should be done on a regular basis for all code. Organizations should also host bug bounty programs to help monitor for potential issues and provide the security community with a well-known reporting path for issues.

There is a perception that security is expensive. While there are some fundamental investments that need to be made in security, often security can be done with the tools that the team already has in place. And with DevOps, it's essential to use the tools that developers are already using. Zero Trust can help demonstrate to executives that the organization is pursuing the most effective strategy to secure the organization.