The Breach: Part 3

by John Hensberger | Oct 18, 2021 | Breach Response, Cybersecurity

The Breach – Part 3

In “The Breach – Part I” and “The Breach – Part 2 ”, Black Bottle IT shares a true story of cybercrime that took place with a manufacturing client and how it unfolded.

“The Breach” continues…

Engaging the Data Breach Coach

It turns out; Data Breach Coaches are law firms that specialize in cyber statutes. It finally felt good to be talking to someone who I thought would guide us through our options. Their role, as it was explained to me, was to quarterback the breach response. They would:

Take the lead and run point
Advise our team on what to do next
Bring in third-party expertise, if necessary, to help us mitigate any legal risk.

This information all sounded great. My first question was, “Where do we start?”

A formal engagement letter was emailed to me shortly after our call ended. I read it and found some language I didn’t like, as with most legal documents. Most things I could live with, except the clause that says, “Data Breach Coach, acting in the best interest of the client, can engage and agree to terms with third party services, with all financial responsibility passed on the client, without client consent.” It says the law firm can spend company money without anyone consenting or agreeing. The second call we would have was to review the engagement letter. When I questioned this language, “only to be told,” the response was, “It’s standard language, for your own good so that we can move quickly.” I had this line removed from the agreement and officially engaged the Data Breach Coach.

Forensic Swat Team

One of the first things the Data Breach Coach advised was to engage a third-party computer forensic company to determine what data was lost when it was lost, etc. We had a call, and the person leading the conversation had a super high sense of urgency. He wanted to send 2-3 forensics technicians on-site the next day. I paused and said, “How much is that going to cost?”

Our insurance coverage was up to 250K of expenses. Anything over was coming from the company coffers. We were a small company; having thousands of dollars in fees would put the company in a vulnerable financial state. So, I was always trying to walk the line of protecting the company, doing our duty to investigate, and being very mindful of the financial situation.

Back to the story – The forensics company said we didn’t have time to wait, which was the best option. I countered with, can’t we gather information with our staff and send you what you need? Let’s start there, make a list, and we’ll begin compiling.

Date of Discovery

All along the way, third-party companies were lining up with their bags wide open, hoping to get them filled with “incident response” money. Most of them were trying to create some real sense of urgency to engage and take action. I never fell for the high-pressure tactics; I wanted to get some options, evaluate the risks, and make an informed decision. However, in the days that followed, the attorneys started educating us in cyber statutes’ generalities. Most of them had a requirement to respond with their timelines after the “date of discovery.” This timeframe was the only timeline that mattered. We had time to rationally engage third-party forensics to identify the scope of the data breach to formulate a responsible response plan.

Continue to Part 4

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.

The Breach: Part 2

by John Hensberger | Oct 4, 2021 | Breach Response, Cyber Insurance, Cybersecurity

In “The Breach – Part I,” Black Bottle IT shares a true story of cybercrime that took place with a manufacturing client and how it unfolded.

“The Breach” continues…

What Do We Do Know?

Immediately following, we start to take action to contain the incident. We immediately block the traffic in/out from the bad actor IP addresses. We look at some simple log data and verify that the Company has been transmitting data to these IP addresses — for as long as we have logged data over 30 days.

In the meantime, I’m getting SOS calls from the CEO, wanting an update from the FBI meeting and wanting to know, “How bad is this?” Since I don’t know the extent of the incident, I don’t have much, other than we think we’ve contained it. He asked good questions: “Don’t we have a firewall, AV, and a 3rd party that is supposed to help us with these things?” All of which the answer is yes. This then follows by the inevitable, “How could this happen?” Someone once told me that when you have bad news, the sooner it’s delivered, the better. So my message to the CEO was, we think we’ve contained it; now we need to figure out what the damage is, which is something that I need to figure out now.

The Response

So, let’s recap:

the FBI informs us that we have been “cyber-attacked” but offers no more information.
We think we’ve stopped it.
Our CEO is asking us questions about whether the company can survive this. And we have no good answers.

It was a pretty bad 24 hours. So, the real challenge for us was, what do we do next? First, we mobilized our third-party MSP to help. They scheduled time over that weekend to visit our location, install some advanced security tools, and “clean” every machine in the building. We didn’t know what had caused the breach, but this seemed like a good place to start.

Next, I got on the phone and started calling companies for advice/help. Most of these calls went something like this: “Yes, we’d love to help you; you need to determine what data was lost. So, you can buy blocks of time for 50K each. When do you want us to start?” Our Company was small, so spending money in chunks of 50K would also very quickly lead to the company’s financial demise. So, after calling and listening to multiple solutions, I had some options. In the background, Company stakeholders are asking me, “How bad is it?” “Are we out of business?”

Ah-Ha. We Have Insurance for That

At some point during the first 48 hours, the CFO had inquired to our insurance agent about having cyber coverage. It turns out we had coverage. She launched a call to our service representative, who then told us, we would receive a call within 72 hours with instructions. I was in the mindset that the company might not survive 72 hours. I didn’t wait around and continued searching to find some resource to help; that made sense. Then, I received a call from the insurance customer service rep. She told me that if I checked the policy, I would find instructions for utilizing the “Data Breach” services that our company was entitled to as part of our coverage. I thanked her, started reading, and found that we indeed had access to a “Data Breach Coach” or a list of them. I picked the first name and called them.

Continue to Part 3

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.

My Breach Story: Part 1

by John Hensberger | Sep 28, 2021 | Breach Hotline, Breach Response

The Breach – Part 1

I’m not sure how many cyber security professionals have been on the receiving end of a data breach and had to navigate their way through it. But, part of my passion for this industry was born in my experience. I was a typical IT Manager, supporting a companies software and infrastructure. They had a few on-site IT professionals, and some outsourced partners, and were focused on growing revenue, operational efficiencies, and doing more with less. The Company had what I’d consider typical security tools in place. A firewall with IP blocking and blacklisting, a modern AV/Anti-Malware agent, email security tools, etc.

The Call

One typical day, I’m at a lunch meeting with the CEO at Panera. I got a call from my IT Manager since I was at lunch with the CEO. I wanted to stay focused on our conversation, so I ignored the call. Immediately following I got an SOS text to call her ASAP. This was not like her, so I knew something needed to be addressed, so I excused myself from the CEO and stepped outside to call her back. I could sense a little panic in her voice, so I immediately asked her “What’s wrong?”. She tells me that they received a call from the FBI, stating that an agent would be on-site the next morning to discuss a cyber security incident and that IT leadership and any 3rd party related to infrastructure should be present. I paused, and being somewhat skeptical, I said, call the FBI office and verify the information provided (thinking it was a scam call). She said, I already did, and it’s real. After returning the table, the CEO asked me “What’s wrong?”, I relayed the information, and his first question was “Is this something that could put the company out of business?”, to which I replied, “I don’t know yet”. This was the beginning of a very long and hectic three months.

FBI On-site

True to their word, the FBI showed up the next morning, and we had all the necessary players around the table to ask questions and determine the scope of the situation. Before we could open our mouths the FBI proceeds to tell us that he’s not even a cyber crime agent, he’s a kidnapping/ransom agent. That he doesn’t know anything about cyber crime, and his job is to read us the information about the incident. So, he begins and reads a document that says the FBI had been monitoring some bad actors in eastern Europe and that our Company was 1 of 30 companies transmitting data to foreign IP addresses. After reading the statement, he gave us the specifics of which IP addresses were in scope. We start asking questions, to which he simply says “My job was to inform you of this activity, if you need more specifics, here is a card of a cyber crime agent that might help you”. We shake hands, the meeting is over.

Continue to Part 2

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.