Episode 21

Unbelievable Lessons with Greg Crabb Former Chief Information Security Officer of the US Postal Service

What does it take to secure 160 million Americans privacy? Greg Crabb, former Chief Information Security Officer at US Postal Service joins Tech Transforms to talk about his experiences from his time as Projects Coordinator for International Fraud to his role in the 2020 US Presidential Election.

Episode Table of Contents

[00:42] Greg Crabb in the House, Founder of 10-8 Cyber
[10:33] Good Guys Get Together With Greg Crabb
[20:05] They Want To Do the Right Thing
[29:32] Greg Crabb Had a Long, Deep, and Intense Service

Episode Links and Resources

Greg Crabb in the House, Founder of 10-8 Cyber

Carolyn: Today, our guest is a rockstar. His background just blew me away. Greg Crabb, founder of 10-8 Cyber and strategic advisor to several organizations, but that doesn't even scratch the tip of the iceberg of who our guest is today.

Greg: Thank you, Carolyn. I enjoy the opportunity to chat.

Carolyn: You recently retired after 20 years with the US Postal Service where you wore many hats. From being a project coordinator for international fraud, assistant director of economic crimes, you ended your career as the chief information security officer and vice president of USPS. That's the very tip of the iceberg of your career. I want to kick it over to you and have you tell us your story.

Greg: The mission of my life has been to protect others and drive benefits for society. I was grateful enough to have the opportunity to retire last year after 30 years of federal service. When I joined the postal service in the mid-90s, I spent the first several years of my career being an auditor. I was responsible for the old electronic data processing controls portion of the financial audit. There I learned an amazing amount of information about how computers work, mainframes, networking, and all that sort of thing.

In 2000, I transitioned to spend seven years investigating the origins of Eastern European organized cyber crime. That was an amazing experience. I got the opportunity to really attack an organized crime group. It was based out of Ukraine and had splinters all over the world. I worked with Europol and Interpol, the Secret Service, the FBI, and many other organizations in between.

Bigger and Better Things for Greg Crabb

Greg: In about 2005, I moved to Washington to take on bigger and better things. Then in 2010, the international supply chain was attacked with some parcel bombs from AQAP. AQAP put PETN, it's the liquid explosive that we all know as why we can't carry water bottles onto airplanes. It completely changed the security model of how international supply chains work for moving parcels. I spent a number of years working with an international community of 190 countries to develop new standards. Worked with civil aviation authorities to properly secure the supply chain from a commercial aviation perspective for parcel security.

In 2014, I got tapped to respond to a pretty significant breach at the US Postal Service. In that moment, I transitioned from being the law enforcement officer who was the hunter to the chief information security officer who was the hunted and responsible for an amazing network. I was grateful to provide security for 160 million delivery points, private communications, and parcels for all of America for six years. As we talked before the show, I was really grateful to have the opportunity to help protect the 2020 election. It was just an amazing collaboration with the folks at CISA and many other organizations across the country in order to pull that off.

Carolyn: Can you talk about what you did to prepare for that? Talk about pressure. You'd been preparing for 20 years.

Greg: I had been preparing. I wouldn’t have been as successful in protecting the technology assets the postal service relied on to move 70 million ballots if I had not had those experiences.

Greg Crabb Is Dealing With Eastern European Organized Cyber Crime

Greg: I’ve been dealing with Eastern European organized cyber crime and understanding what it takes in order to protect a network from everything. From disinformation campaigns to all of the technical details that are necessary to secure a network. Where do you want to start? From an application security perspective for the organization, from an OT security perspective, all of the technology that was necessary in order to be able to move the ballots were in the consideration.

Carolyn: It's interesting because when I think of the US Postal Service. I'm not going to lie, it's a little boring.

Mark: You think of physical security, OT.

Carolyn: It's a letter and you're moving a physical thing. But like all the cyber that was involved in it, is it the processes that you were securing, the databases?

Greg: It was everything. From the delivery scanner, the letter carrier that is driving by your house every day, securing that scanner to know that your ballot was in the mail, to all of the operational technology that exists. Now, imagine big warehouses larger than big football stadiums, where mail processing is made. Huge pieces of equipment were down to your address. All the packages come in and we sort them down to each delivery point.

In total, I was responsible for the protection of 1.2 million technology assets. Petabytes and petabytes of data relative to package tracking and those kinds of things. Only 630,000 employees are necessary in order to be able to deliver that mission and interact with all those technology points.

Mark: That's just at the federal level. I can only imagine the collaboration across states and everything must have been massive.

A Massive Enterprise Focused on Collaboration

Greg: I did not have the responsibility of dealing with each of the states. The postal service, in order to be able to secure the election, is a massive enterprise focused on all of the collaboration, developing standards on how mail pieces are supposed to be formed. There's a lot to make sure that everything gets delivered on time. If the ballots aren't there, they don't get counted. That was something that not only my role as chief information security officer but my security partner, who was the chief postal inspector, was out all over the country in facilities. Basically, just making sure that each of the communities were getting their ballots in a timely manner from all of our delivery operations.

Mark: I know you've done a lot of work with cyber crime and you've worked with adversaries, I guess, as well. I’m curious to know how you've seen this whole landscape transform over the years.

Greg: I started working in organized cyber crime investigations in 2000. I was asked to help the FBI with a case coming out, or some significant amount of fraud against eBay, coming out of Ukraine. It took me several years to wrap my arms around it. I was ultimately able to arrest a number of folks from Ukraine and other Eastern European countries that were involved in this. But really, I had the opportunity to sit down and talk with a lot of investigators from Eastern European countries.

We had a conference in Warsaw a number of years ago. These were early days. They talked about how they started to see car smuggling gangs that were based in Eastern Europe starting to have technology equipment in the vehicles where they were getting arrested.

Good Guys Get Together With Greg Crabb

Greg: Eastern European car smuggling gangs would basically get caught in Poland with stolen cars. They popped the trunks and there would be a bunch of technology equipment, credit card skimmers, and other technology that's necessary in order to be able to commit those types of crimes.

There was a conference. Good guys get together and do the International Association of Chiefs of Police Conference and those types of things. But in 2001, there was a conference in Odessa where a bunch of criminals got together. They referred to it as, "The First International Carders' Conference." The members of that meeting became my targets.

Carolyn: How did you find out about it?

Greg: I had the good fortune of one of the criminals that I was investigating had hacked into a server in San Jose, California. At the time, I was based in Northern California. The data center was on my drive into the office. He was sending all of his communications through that server. I intercepted 40,000 of his email messages.

Carolyn: How did you find that?

Greg: Long story. It was just tracing IP addresses and getting back to the source of where all of his communications were coming from. They all sourced to this particular server. The company was called Hurricane Electric in Fremont, California. I worked with the US Attorney's Office and the victim to get approval to go in and review my suspect's messages on a daily basis. So I had this unbelievable wiretap on this criminal and all of his email messages. Not only facilitating his crime but with all of his cohorts in crime who were doing these activities.

Greg Crabb With Eastern Law Enforcement

Greg: One thing led to another and I learned about this International Carders' Conference based in Odessa and basically focused on this group. That led me to the opportunity to really work with Eastern European law enforcement officers. I had occasion to work with the Russian FSB and the Ukrainian MVD, and even law enforcement officers in Belarus. I proudly display the hat there from my colleague in Minsk, Belarus. Unfortunately, he was actually arrested for working with me. We went to the country in '05. The suspect had a website. The banner on the website said that their objective was to take the United States back to the time of 1929.

Fortunately, the Belarusian law enforcement officers were willing to work with us until the government got involved after we left. From the computer equipment in that case in Minsk, we recovered over 55,000 full infos. These are victims in the United States where they've got their mother's maiden name, their social security number, all of the answers to the questions that are necessary for those knowledge-based questions. He had hacked into LexisNexis, and his hack was actually the subject of a congressional debate. The subject's name was Sergey Pavlovich.

It was interesting to be able to work with all of those folks and deal with not only, on a personal relationship, the police officers but then the governments from a not-so-friendly perspective. I was never invited back to Belarus again, I was stupid enough to try to go back to Belarus. But I think the US embassy was smart enough not to let me in. Those experiences were foundational for me to understand what's necessary in order to be able to counter what we see from a law enforcement perspective.

What James Woolsey Taught Greg Crabb

Greg: Like James Woolsey taught me, there's little difference between a Russian businessman, a Russian politician, and a Russian organized crime figure. They're one and the same, like the people that organized that International Carders' Conference in Odessa, I look at where we are in the political world today with Ukraine and Russia, and the United States. And I started going to Kyiv in November of 2003, trying to get the guy's name was Dmitry Golubov. Dmitry Golubov arrested.

He was one of the key organizers of this conference, responsible for massive amounts of fraud against financial institutions and online companies. It was very difficult. He was protected by the police in Odessa. They were on the payroll. It really came down to the Orange Revolution. In early 2005, late 2004 to 2005, there was a highly contested election in Ukraine. The first election was called by the international community to be fraud, and a second election was made. It was very close from a decision perspective.

You might remember that the candidate that won was actually poisoned by the Russian FSB in Switzerland while he was traveling there. He won. When he took power, the Ukrainian MVD asked for me to come over and actually present my case. I got an opportunity to go over and brief my case to the Minister of Interior to Ukraine. Two weeks later, they arrested Dmitry Golubov.

Mark: In London, the UK?

Greg: No, it was in Ukraine. It was in Odessa. I do stop for a second. The Orange Revolution was extremely important in Ukraine. It was really that turning point in the history of Ukraine where they went Western leaning, they were looking to democracy.

Why the US Embassy Was Giddy When Ukrainians Invited Greg Crabb

Greg: They were looking to, how can they become more westernized? At the time, when the Ukrainians asked me to come over, the US embassy was giddy. We actually have a government that's interested in working with the US.

I remember going with the special agent that was the FBI leg-att there in Kyiv, Ukraine. His name was John Boles. We drove over to the briefing with the Minister of Interior to Ukraine. He was so excited that we were going to present this case. Then for them to actually, the Ukrainians, to go and arrest Golubov was unbelievable. I was invited back over after the arrest to again do something that had never been done before. Boles was really excited.

I was the first US law enforcement officer to ever be asked to interview a Ukrainian on Ukrainian soil. Albeit, we went to jail. He told us to buzz off, but so be it. It was a great turning point in our relations. Now when you see what's going on in the international community, I can only pray for the folks in Ukraine to be able to maintain their democracy and keep the coalition here, or the West to be able to help them.

Mark: Given all of the things that are top in the news right now, this is really interesting. Ukraine is in the news every day.

Greg: Unfortunately, we weren't able to present our case in Ukrainian court against Golubov. I won't say it was corruption, but after some time the case was dismissed. Several years later, Dmitry Golubov actually became a member of Ukrainian parliament. And you're just, "That's the way the world works, people."

Carolyn: The James Woolsey quote holds true.

They Want To Do the Right Thing

Greg: Exactly. You know it. I saw it firsthand. I've seen it again and again in my interactions with folks that are over in Eastern Europe. On an individual level, I was able to form some amazing personal relationships with law enforcement officers. They want to do the right thing, but governments, politics, and corruption are difficult things to overcome. I think that's where we really need to continue to focus and understand. Thank you for letting me take a little walk down history lane. I think all of those lessons are important today to understand what we face from an organizational national security perspective for the country.

Mark: When I think of cyber crime, cybersecurity, I think of technology, bits, and bytes. I think of leveraging cutting-edge kinds of technologies and the way people do what they do. I'm really curious to know because you've talked about a couple of things which makes me think really just grassroots intel, spy versus spy. How much of this world is HUMINT or human intel as opposed to the technology piece?

Greg: Technology's just an instrument to the motives of the actors. I think that one of the things that, as we look forward and we look at the attacks like SolarWinds and you name the Eastern European-based attacks that we've seen, they're all motivated by the objectives of either greed or control. Or I often recall, and this is a supply chain story. It's going to start a little weird. One of my colleagues in Eastern Europe always used to say, "Mr. Greg Crabb, trust no one. Not your wife, not your girlfriend and not your lover."

Supply Chain Related Story

Greg: How could that possibly be a supply chain-related quote? I knew all three of his, by the way. His wife, wonderful mother of his children, awesome. Girlfriend was just beautiful. His lover, she was smoking hot. But what he would always talk about is that from a mass surveillance society, you need to understand your relations. You need to understand your most trusted intimate relations in order to keep them controlled.

I think when we see the types of attacks with SolarWinds and the other supply chain attacks that we're seeing in software development lifecycle, we need to understand that mass surveillance is a technique that is used in those cultures. We need to understand and account for it in our information security practices. There's bits of the story that I've left out there, but you can get the main point.

Carolyn: I feel like we're...