DevOps: Pushed by Circumstance or Pulled by a Dream, with Tracy Bannon

Tracy Bannon Senior Principal / Software Architect & DevOps Strategic Advisor at MITRE and ambassador for the DevOps Institute talks through the original DevOps timeline. Join as Carolyn and guest host Steve Mazzuca find out what happens when Dev fraternizes with Ops.

Episode Table of Contents

• [00:48] DevOps Strategic Advisor and Ambassador
• [10:34] Respected DevOps
• [18:35] The DevOps Pipeline
• [24:05] DevOps Institute

Episode Links and Resources

• DevOps Institute
• Project To Product

DevOps Strategic Advisor and Ambassador

Carolyn: Today, I have Steve Mazucca or The Mas as I like to call him, co-hosting with me. It's always fun to have a conversation with you Steve. The hard part is going to be getting you to be quiet, so we can get our guest Tracy Bannon. He is Senior Principal, Software Architect and DevOps Strategic Advisor at MITRE, as well as an ambassador for the DevOps Institute. So welcome Tracy.

Tracy: I'm thrilled to be here today. It's always fun to have these conversations.

Carolyn: You are a striking woman with pink hair and you were in development, which makes you in my mind, kind of a unicorn. I would really love to hear your story.

Tracy: I'll start with the pink hair and go backwards from there. I've had little bits of color in my hair for years. My mom was an art teacher. My dad's more on the math and the sciences side of it. I kind of have that left brain, right brain, need to express myself. Over probably the last two or three years, as I've been doing more remote work, I was having more fun with the pink and decided that it's the pandemic. Let's stretch things a little bit more. I'm just loving it. So that's a little bit about that piece of it.

But as for me being a woman in technology, I actually like to come at it in reverse. To say that I'm a real technologist and not say I'm a woman technologist. It matters, but it doesn't matter. What's important to me, is I've always been so interested in tech.

A Woman Developer

Tracy: Someone asked me, "When was the first time you realized that you liked computers and that you were into computers?" It's a long story, but I'll make it very short. I can remember building a computer out of a box and cutting and putting mag tapes on the outside. Yes, I just told you how old I was. And arguing with my brother on who got to sit inside it and be the brains. So I remember being real and I couldn't read yet. I remember that very vividly. It goes a long way back.

Carolyn: Did you end up being the brains?

Tracy: Yes I did. I happened to be a little bit bigger than him. Even though he's two years older, I happened to have the weight advantage. As for being a woman developer, I've always been in tech. I never thought anything about the makeup of the team. That’s because I always tagged around with my older brother and his buddies. I considered myself one of the guys. One of the gang would be a better way to put it. I realized about midway through my career that there was a little bit of uniqueness to it. As I would look around the room, I would be the only woman on the team.

Now, occasionally there would be fantastic women involved, more on the database side of things, who had grown into that. Very few from a development perspective. We did see some spikes in industry, we saw that. But we're seeing that decline recently. But across my career, I tend to come at it that I'm a technologist. If you need to give me an adjective, make it real, instead of woman. But that's a little bit about me.

A Technologist at DevOps Who Happens to Be a Woman

Carolyn: I love that you want to take the emphasis off women. That you're a technologist, you're a developer and you happen to be a woman. You're often the only woman in the room. I'm often the only woman in the room and it will be a room of many people. But I do love that you've always just thought of yourself this way. What was your first development? Well, what was your first job actually?

Tracy: First paying job, was actually a lifeguard, but we won't go that far back. If we go forward, I worked throughout college in different corporate settings, always related to technology. When I graduated, I actually was independent before anybody was doing anything independent. And then it happened on my way into the engineering department at AccuWeather. My husband and I were already married and he was in operation. You can tell there was a little bit of, hey, take a look at this resume. That helped get me in the door in a very heavily male-populated tech group.

Yes, we were both within AccuWeather. That's actually a segue to a fun story. He's in operations, I'm in engineering. You could actually say that we're the original DevOps because we've been married for a couple of decades now before we said DevOps. So something would happen with engineering and they would call because there was a production problem. They would call him and he would realize that we had talked about something that week. Dinner time chatter.

Debugging Things Together at DevOps

Tracy: He'd say, "Didn't you guys roll out a change to XYZ." I say, "Oh yes." "Can you open up a window?" It wasn't a browser at that point. "Can you open up a window, let's take a look. We need to look at this queue or this record or this log." And we would end up debugging things together. They started to no longer just call and ask for him. They’d call the Bannon house because they would get both of us to solve a problem, it was really cool.

That showed me how important it was that I was writing software that could be operated, could be managed, could be maintained, could be debugged. It also taught him how important it was to give me access. I didn't have to have right access. But I needed to be able to check the cues. I needed to be able to look at these different things. That started me having that fraternizing with Ops. I've always been an advocate of having Ops at the table, even before. Well, before we coined that wonky phrase DevOps.

Steve: When did we start coming out with that? Is that Gene Kim, is he famous for that? Or is that even before Gene Kim, as far as when he started coming out with that?

Tracy: It was before Gene, but I cannot remember the fellow's name off-hand. I'd have to Google it. It strikes me that it's Patrick something. It wasn't that long ago. Maybe in 2011, 2012, somewhere in that, he made a comment about it. It really resonated with folks.

An Evolution

Tracy: Gene was brilliant to realize that it's a true and valuable story. He started really to table pound to get the message out and started this. I'll call it an evolution. I hate the revolution thing. But he started us evolving and thinking about it.

Steve: It's certainly come a long way. You and I obviously have done a lot of work together over the years and a couple of different iterations across government. I've only known you supporting the government, but I know you did some other things before. How long have you supported the federal government? I don't actually know that story.

Tracy: Federal 2015. Not that long, not an entire career focused on the federal government. But I was working across state governments starting in 1999.

Carolyn: Is it really, really different to be a state versus federal?

Tracy: There are parallels and there are differences. In the federal government, we have a mandate on citizenship. You don't have the same mandates for a state-level data center. I can have a foreign nationals, I can have different types of folks coming in with different types of visas, to be able to support that. That's starting to grow and evolve and change. The policies are much stricter at the federal level and the size.

Some of the biggest states are similar to some of the smallest federal agencies. But think about the economies of scale. It's just that much bigger at the federal level. In defense, not a whole lot of defense at the state level. That's probably the most interesting, different mission that I've been involved with.

Respected DevOps

Tracy: Being involved on the public sector side of the government, the civilian side of the federal government, that's where Steve and I got to know each other. It was with the IRS and treasury and around that side of it. Changing the focus more into defense has been very humbling just because of the sheer complexity. Think about NATO with 30 different nations involved in technical decisions and discussions and the complexities that come with them. It's just mind-boggling.

Carolyn: Just think of communication alone. Being able to communicate across all those different groups, it breaks my head. Can you think of any use cases between state government, federal government vice versa? A best practice that one should pick up from the other.

Tracy: Leading practices abound across both of them. Whether it is with respected DevOps, whether it's with respect to leveraging AI and ML, to improve. One of the things that I'm seeing in both places. I experienced this first with the state of Colorado and that was embracing Cloud. This was multiple years ago, they looked at it and this was one particular agency. They said, "In order for us to be secure enough, we actually need to go to the Cloud." Now that seems to be the opposite of what people think about right now.

Oh no, we don't want to go to the Cloud. That could be a breach. But they looked at it from an economics perspective and said, "For us to have the same number of professionals, with the same level of training, with the same SLAs, with the same contractual obligation to keep us safe, would cost us this many millions. Hundreds of millions."

A Part of the Opportunity

Tracy: Where if I have that through my contract with a provider like Salesforce or Service Now, or AWS, or Google, any of them, I have that out of the box. I have that as part of it. For the most part, I have that as a part of the opportunity. That's one thing that I saw the states do a little bit before the federal government. I’d say the federal government is tighter in its cyber practice, absolutely. Tighter in its cyber practice, but it's like any technology. It's not about the tech, it's about how we apply it. So how about we go about problem-solving. We do a lot of things in the commercial space. I did a lot of commercial work.

The commercial is a little less fettered. They're not as tethered to what the policies and the bureaucracy can be. Federal bureaucracy is a little more impacted by the administration changes. Trying to think of what the acronym was the other day. Somebody was joking with me. Instead of meantime to recover MTTR as a DevOps, it was a meantime to command change. It was MTTC or MTCC. Because you can figure out how much disruption was going to happen, and that's something very different.

The states have much more continuity with their technology leadership in specific, than I see in the federal space. The defense definitely, because of the time you do rotations, you may be there for six months, a year, or two years. You see it more in this less civilian side, less frequently. But still the federal is much more changeover, than I see at the state level.

The Challenge With DoD

Carolyn: The defense, Mas, that's your world. I could see a little bit of two things with that command change. Fresh eyes, means fresh ideas. Also, it feels like it could be a huge setback.

Steve: The other challenge with DoD that we run into is it's everything about the network edge security. Everybody is commercial and never gone in civilian, not all a bunch of places, but the vast majority is civilian. We're using the public internet, we're using everything that's out there. They have access to everything that is commercially sound. In defense, it starts with, we don't have what we don't have access to. You start with what you don't have access to and you've got to build off from there.

The problems are obviously incrementally different. But the problem is also we can't take advantage of a lot of things that we can. We have to constantly fight that battle on both. As technology advisers, we have to go in there with that understanding that we're talking about solutions and products and technologies that sometimes you can't utilize all of the capability. Can we utilize enough to actually make a difference in your mission? Even beyond the fact that the kernels aren't going to be rotating in and out correct.

Tracy: It's interesting. There's always an opportunity for those fresh eyes to come in and infuse a lot of new energy and thought. But that depending on how large a program is, it takes them maybe a year to 18 months, to really hit stride. And so, there's a lot of churn that can happen during that time. Sometimes you'll see that folks will entrench themselves like, oh no, there's a change of leadership coming. I'm going to stay the course.

Specific Thing About DevOps

Tracy: While there's a leader that's talking about this amazing different type of infusing, of innovation, those in the trenches are saying, "Okay, I'm going to stay the course until we figure out how this solidifies, or how it plays out a little bit more." It causes a little bit of tension back and forth. But Steve, you bring up a really good point. I think about DevOps in specific. On the defense side, I jokingly say that they like the term DevSecOps.

But it's really DevSec, pause, wait, there's some other things going on, then SecOps. So the idea that we see of that infinity loop, right the figure eight, for there to be constant and continual feedback from the war fighter, from the constituency, to the developer. It's more difficult because there are different groups that are in charge of it.

I don't mean a guy in the other room is in charge of operations, versus me here in charge of Dev. They may be contractually a different part, they may be totally a different part of the service. Think about deploying onto a Naval vessel. Ops, there is a lot different than Ops that would be CONUS. Continental US sitting right beside me. It does have some pretty different challenges and is not insurmountable.

There's so much goodness that we should be looking at North Star to Steve's point. What's happening in commercials and then, what's my problem? What is the actual thing I'm trying to do? Then looking at commercials and saying, "What applies here? And what could be tailored? What could be improved and brought in?"

The DevOps Pipeline

Tracy: As opposed to, I'm going to go do what they did over here. I always use the example and it's trite at this point because so many people have heard it. But we're not just going to Netflix this. It's not just 50 releases a day. If I'm putting software onto a tank, if I'm putting it onto a jet, I'm not going to release 50 times a day. I may release a couple of times a week, but not 50 times a day. And so, there's economies of scale and things to learn with that.

Carolyn: I want to go back to that feedback loop that you talked about. Have you seen an agency that does that really well or a group? We don't even have to name names. Because just what you said, putting those software releases onto a tank, where we're talking about lives on the line. You need feedback from the guys operating it. Have you been in situations where that's just really smooth and they figured it out as a well-oiled machine?

Tracy: Yes, at scale. Not as much, but definitely there're amazing pockets of goodness. If I think about the things that are happening within the Air Force, if I think about things that are happening within the Navy and specific things with the Space Force, there are some fantastic loops that are going on. It depends on what the type of software is.

Also, it depends on what has to be done for fielding it ahead of time. There're policies and procedures that are in place, that would say I can't take that code that Tracy wrote. That she committed, that it was unit tested, that it automatically went through all of those things that we think of as the DevSecOps pipeline.

An Operational Fielding Exercise

Tracy: It actually has to go through an operational fielding exercise, before it could actually go into a war protection type scenario. Think of it like when people talk about the sequence and getting to pre-production. A lot of things happen in a beautiful figure eight up to this pre-production. Then there's one additional step, which is to live in the field. It's almost as though you have the figure eight and then another little punctuation off to the side. Steve, is that your experience too?

Steve: I think so. You differentiate when we think of war fighters, we always go right away to the weapon system to the plane or the tank. And now those software engines are a little bit different than all of the other business systems that are still out there. The vast majority of software being developed for them is still through the software batteries. It’s really more about the business logistics of doing everything else. Not necessarily command and control that the missile fly straight, that's being done at a very discreet kind of lab oriented.

But everything else, which at least looks on the surface anyway, with all of these software factories, the government is trying to move faster in that it embraces the CIT, the pipeline, and does more things in the Cloud. So yes, I'm encouraged by it. There's still a little bit of a disconnect between the CIC of the pipeline and then the ATO process. There is always this big Cloud around the ATO process, which does put a monkey wrench into things. Because every time you change any aspect, could we have broken something that could cause a security vulnerability?

The Purpose of Doing DevOps

Steve: How do we get around that? How do we make that faster?

Tracy: There's so much goodness that's happening now, to focus on CATO, Continuous ATO, the authorization to operate. It's a good debate on how real that CATO is, the ATO process as well as a platform that's underneath it. Then the thing that you need to look at and audit and be super focused on, is what's moving across the top of the Delta and the change? But that means that your pipeline needs to have a tremendous amount of auditability. Instantaneous audit ability throughout that process.

The RMF process in and of itself, is a good and strong framework. What's difficult, is helping the cyber professionals become part of the earlier parts of the design. I did a Navy project, I guess this was about two and a half years ago. And I really learned so much about the RMF process during that.

RMF, it's a Risk Management Framework. It is a way that you assess and evaluate a project, or a system, or a product before it goes to production.

Normally, the feed into RMF is that you have all of the designs complete, all of the boundaries, all of the information flows, everything complete. I thought I had a brilliant idea that I said, "Okay, guys, I've been hanging out during DevOps things. Not for the purpose of doing DevOps, but I've been leveraging those capabilities for many years, to take systems into production." I have horizontal teams, I want transparency for anybody who's involved in this. Hey, you RMF guys, those folks are going to work on our ATO, come on...