The Breach: Part 4

by John Hensberger | Oct 25, 2021 | Breach Hotline, Cybersecurity, Risk Management

The Breach – Part 4

And the Story picks up where we left off here.

Date of Discovery

All along the way, third-party companies were lining up with bags opened, hoping to get them filled with “incident response” money. Most of them were trying to create some real sense of urgency to engage and take action. I never fell for the high-pressure tactics; I wanted to get some options, evaluate the risks, and make an informed decision. However, in the days that followed, the attorneys started educating us in the generalities of cyber statutes. Most of them had a requirement to respond with their timelines after the “date of discovery.” This was the only timeline that mattered. So, we had time, but much, to rationally engage third-party forensics to a specific scope of the data breach so that we could formulate a responsible response plan.

250 Records Lost – What A Relief

After sending the forensic team copies of the hard drives from machines in scope, we had many calls with the team. After their investigation, they thought we could have lost about 250 records of personal information. At first, it was a relief, as it could have been much worse. But shortly after, I started to realize that their analysis was speaking in hypotheticals. The words beings used were “we think,” “they might have,” “it possible,” rather than more explicit language. So, I started to understand that they didn’t know, but the forensics team advised me on the potential risk of what was accessible to the cyber attackers. After this call, we had a pretty good idea about the size and scope of the attack. At this point, I was finally able to provide some tangible details to the stakeholders of the company. Until now, the information I had was all hypothetical; now, we had some excellent news to act on.

The First Invoice — Yikes

A few weeks into this saga, the Company received its first invoice from the Data Breach Coach (attorneys) and the forensics company. Let’s back up. We engaged this firm because we had access to them through our insurance coverage. And, we received about 60% more financial relief from all the expenses if we used the insurance companies’ providers rather than just find our resources. So, naturally, we engaged the providers recommended by our insurance company. But, there was a limit to what the coverage would allow for. At first, I thought the coverage was way more than enough until the first invoice arrived. The hourly billable rates were so high, and I felt they misplaced a decimal point. Our first invoice ate up around 35% of our allowed coverage. I needed to make sure the Company was using its coverage limit wisely. The Data Breach Coach and forensics team afforded plenty of opportunities to do work and eat up our remaining coverage.

The Response

After the scope of the event had been determined, it was time to work on the response. The legal team briefed us on the types of actions that we needed to take. First, we needed to determine what states the impacted parties were residing in. Each state has its reporting requirements when a data breach involves personally identifiable information. Some states even require credit monitoring and other services to protect individuals from identity theft.* I quickly realized that the attorneys understood these requirements and would be an extremely valuable resource. They took on crafting notification letters to 15+ states on our behalf and having guidance that his juncture of the story was very comforting. I understood our risk and felt comfortable we were doing the right things.

*note — These requirements are now commonplace in most states, but during this time, these requirements were not the norm

Our MSP Should Know How To Help Us Remediate… Not So Fast

At the same time our talks with the attorney and forensics, the Company was actively trying to remediate the root cause of the cyber attack. We were confident we had stopped the attack. But, verifying we had no lingering effects of our attack proved harder than we thought. We reach out to our MSP for advice and assistance. We planned a weekend to come in, install advanced cleaning tools, and clean every machine in the building (over 100). We only completed the task to find the infected device (that was cleaned) with some unrelated malware. We scheduled another weekend to re-clean all the machines again. After the 2nd round of cleansing, the malware was found again. It was then I realized our MSP was not equipped to handle our situation. To date, they have served us well, but the problem overmatches us. In summary, they were not security experts, so we needed additional support.

The moral of this story, Managed Service Providers, are good at traditional things, procurement of new hardware, architecting new infrastructure, end-user support.. etc. But, cyber security experts were not. Through lots of activity and a few 3rd parties, we could get our environment clean and remove any remnants of the attackers.

Final Part 5 is coming soon.

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.

The Breach: Final Chapter

by John Hensberger | Oct 25, 2021 | Breach Response, Cybersecurity

The Breach: Final Chapter

Why Does The Attorney General of North Carolina Care About Our Data Breach?

Our Data Breach Coaches had filed the necessary communications with several state Attorney Generals. We were advised that most states would acknowledge, and it would end there. However, the Attorney General’s Office in North Carolina reached out and requested more information. This was surprising to me, as I didn’t know how to respond or why they would be asking for more details. Fortunately, our Data Breach Coach fielded the request, and we had a call to discuss the strategy of how to respond. Our response was most reassuring that we had taken the necessary measures to prevent an issue like this from happening again. This was just another point, where had we not engaged the right people, we would have had no idea the reporting requirement existed and/or how to respond to requests for more information. As I learned, each state has different reporting requirements, and North Carolina was not the last state to request more information.

Credit Monitoring, Makes Sense

After two months, we were rounding the home stretch of our incident response. The final piece was to figure out how to protect the impacted individuals from any potential damages. Our Data Breach Coach asked us how we wanted to proceed. (Let’s pause for a second, this is a common theme throughout the process. At various points, third parties who had expertise in a subject matter would ask us our thoughts on how we’d like to proceed. We wanted guidance from the experts in areas we needed; I thought it strange we were asked this, rather than just presented options, back to the story) Obviously, we didn’t know how to respond, so we asked for options.

As a sign of “goodwill,” they suggested offering everyone credit monitoring for two years. That seemed reasonable to us. So they referred us to a national brand that provides these services. During our first call, they offered credit monitoring, explained the process. All of which was acceptable, but then they started talking about setting up a call center, handling questions by the impacted individuals, printing notification letters, and lots of other services. All of which sounded like a great idea had the scope of our incident warranted. However, our situation was so small that it seemed like overkill and yet another way to spend the Company’s money needlessly. So, we set up the necessary processes with internal staff, contracted to offer the credit monitored, and sent out the notification letters. Materially, the incident response was effectively over.

You Want A Detailed Recap Of The Breach Response. Are You Sure?

Finally, after all notification requirements were handled, credit monitoring handed out, and all inquiries were answered, we had a chance to exhale a bit. The incident could have been much, much worse. But we’d made it through. After thinking about it, I realized I didn’t have a formal recap of our response from our Data Breach Coach. I thought, should any inquiries be going forward about how we responded, I should have something to prove that we satisfied all the legal requirements and acted ethically. So, I launched a call to our Data Breach Coach. Out of everything we experienced, this final chapter is something that surprised me the most. After a few days, the Data Breach called me back and acknowledged my request.

Here is our dialog:

Data Breach Coach: What do you want a formal recap of all the activity that occurred on your behalf in responding to this incident?

Me: “Yes, I do; I want to be able to produce some formal documentation of how we responded to our incident in the event it is needed in the future.”

Data Breach Coach: “Are you sure you want this?”

Me: “Yes, I’m sure.”

Data Breach Coach: “Are you really sure?”

Me: “Rather than ask me if I really want it, can you just tell me why you are asking if I want it?”

Data Breach Coach: “If we produced it, it becomes a legally discoverable document, which might not be in your best interest, should future legal action be taken against you and your Company.”

Me: “So, let me get this straight, we spent three months and 250K responding to this incident. We dotted every ‘i’ and crossed every ‘t.’, But it’s not in our best interest to have a formal recap of the actions taken, to prove that we handled the incident in accordance with all the requirements?”

Data Breach Coach: “We can add a high-level summary to your file, should you need it, you can call our office, and we’ll produce it, but having a detailed document isn’t in your best interest.”

Me: “Ok, if that’s what you think is best, then I guess I don’t want it.”

This last interaction left me a little dumbfounded. But in hindsight, I should have expected it. The whole experience was foreign to the Company and me. In the end, we spent our entire amount afforded to use by our insurance coverage to respond to this incident for the potential loss of 250 records. During the initial days, there was lots of confusion, unknowns, and decisions that needed to be made. We didn’t always make the right ones, but we did have enough presence of mind to slow the situation down, get as data as we could, and make informed decisions.

I’m hopeful that our story can help, inform or at least mildly entertain anyone reading.

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.

The Breach: Part 3

by John Hensberger | Oct 18, 2021 | Breach Response, Cybersecurity

The Breach – Part 3

In “The Breach – Part I” and “The Breach – Part 2 ”, Black Bottle IT shares a true story of cybercrime that took place with a manufacturing client and how it unfolded.

“The Breach” continues…

Engaging the Data Breach Coach

It turns out; Data Breach Coaches are law firms that specialize in cyber statutes. It finally felt good to be talking to someone who I thought would guide us through our options. Their role, as it was explained to me, was to quarterback the breach response. They would:

Take the lead and run point
Advise our team on what to do next
Bring in third-party expertise, if necessary, to help us mitigate any legal risk.

This information all sounded great. My first question was, “Where do we start?”

A formal engagement letter was emailed to me shortly after our call ended. I read it and found some language I didn’t like, as with most legal documents. Most things I could live with, except the clause that says, “Data Breach Coach, acting in the best interest of the client, can engage and agree to terms with third party services, with all financial responsibility passed on the client, without client consent.” It says the law firm can spend company money without anyone consenting or agreeing. The second call we would have was to review the engagement letter. When I questioned this language, “only to be told,” the response was, “It’s standard language, for your own good so that we can move quickly.” I had this line removed from the agreement and officially engaged the Data Breach Coach.

Forensic Swat Team

One of the first things the Data Breach Coach advised was to engage a third-party computer forensic company to determine what data was lost when it was lost, etc. We had a call, and the person leading the conversation had a super high sense of urgency. He wanted to send 2-3 forensics technicians on-site the next day. I paused and said, “How much is that going to cost?”

Our insurance coverage was up to 250K of expenses. Anything over was coming from the company coffers. We were a small company; having thousands of dollars in fees would put the company in a vulnerable financial state. So, I was always trying to walk the line of protecting the company, doing our duty to investigate, and being very mindful of the financial situation.

Back to the story – The forensics company said we didn’t have time to wait, which was the best option. I countered with, can’t we gather information with our staff and send you what you need? Let’s start there, make a list, and we’ll begin compiling.

Date of Discovery

All along the way, third-party companies were lining up with their bags wide open, hoping to get them filled with “incident response” money. Most of them were trying to create some real sense of urgency to engage and take action. I never fell for the high-pressure tactics; I wanted to get some options, evaluate the risks, and make an informed decision. However, in the days that followed, the attorneys started educating us in cyber statutes’ generalities. Most of them had a requirement to respond with their timelines after the “date of discovery.” This timeframe was the only timeline that mattered. We had time to rationally engage third-party forensics to identify the scope of the data breach to formulate a responsible response plan.

Continue to Part 4

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.

The Breach: Part 2

by John Hensberger | Oct 4, 2021 | Breach Response, Cyber Insurance, Cybersecurity

In “The Breach – Part I,” Black Bottle IT shares a true story of cybercrime that took place with a manufacturing client and how it unfolded.

“The Breach” continues…

What Do We Do Know?

Immediately following, we start to take action to contain the incident. We immediately block the traffic in/out from the bad actor IP addresses. We look at some simple log data and verify that the Company has been transmitting data to these IP addresses — for as long as we have logged data over 30 days.

In the meantime, I’m getting SOS calls from the CEO, wanting an update from the FBI meeting and wanting to know, “How bad is this?” Since I don’t know the extent of the incident, I don’t have much, other than we think we’ve contained it. He asked good questions: “Don’t we have a firewall, AV, and a 3rd party that is supposed to help us with these things?” All of which the answer is yes. This then follows by the inevitable, “How could this happen?” Someone once told me that when you have bad news, the sooner it’s delivered, the better. So my message to the CEO was, we think we’ve contained it; now we need to figure out what the damage is, which is something that I need to figure out now.

The Response

So, let’s recap:

the FBI informs us that we have been “cyber-attacked” but offers no more information.
We think we’ve stopped it.
Our CEO is asking us questions about whether the company can survive this. And we have no good answers.

It was a pretty bad 24 hours. So, the real challenge for us was, what do we do next? First, we mobilized our third-party MSP to help. They scheduled time over that weekend to visit our location, install some advanced security tools, and “clean” every machine in the building. We didn’t know what had caused the breach, but this seemed like a good place to start.

Next, I got on the phone and started calling companies for advice/help. Most of these calls went something like this: “Yes, we’d love to help you; you need to determine what data was lost. So, you can buy blocks of time for 50K each. When do you want us to start?” Our Company was small, so spending money in chunks of 50K would also very quickly lead to the company’s financial demise. So, after calling and listening to multiple solutions, I had some options. In the background, Company stakeholders are asking me, “How bad is it?” “Are we out of business?”

Ah-Ha. We Have Insurance for That

At some point during the first 48 hours, the CFO had inquired to our insurance agent about having cyber coverage. It turns out we had coverage. She launched a call to our service representative, who then told us, we would receive a call within 72 hours with instructions. I was in the mindset that the company might not survive 72 hours. I didn’t wait around and continued searching to find some resource to help; that made sense. Then, I received a call from the insurance customer service rep. She told me that if I checked the policy, I would find instructions for utilizing the “Data Breach” services that our company was entitled to as part of our coverage. I thanked her, started reading, and found that we indeed had access to a “Data Breach Coach” or a list of them. I picked the first name and called them.

Continue to Part 3

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.

My Breach Story: Part 1

by John Hensberger | Sep 28, 2021 | Breach Hotline, Breach Response

The Breach – Part 1

I’m not sure how many cyber security professionals have been on the receiving end of a data breach and had to navigate their way through it. But, part of my passion for this industry was born in my experience. I was a typical IT Manager, supporting a companies software and infrastructure. They had a few on-site IT professionals, and some outsourced partners, and were focused on growing revenue, operational efficiencies, and doing more with less. The Company had what I’d consider typical security tools in place. A firewall with IP blocking and blacklisting, a modern AV/Anti-Malware agent, email security tools, etc.

The Call

One typical day, I’m at a lunch meeting with the CEO at Panera. I got a call from my IT Manager since I was at lunch with the CEO. I wanted to stay focused on our conversation, so I ignored the call. Immediately following I got an SOS text to call her ASAP. This was not like her, so I knew something needed to be addressed, so I excused myself from the CEO and stepped outside to call her back. I could sense a little panic in her voice, so I immediately asked her “What’s wrong?”. She tells me that they received a call from the FBI, stating that an agent would be on-site the next morning to discuss a cyber security incident and that IT leadership and any 3rd party related to infrastructure should be present. I paused, and being somewhat skeptical, I said, call the FBI office and verify the information provided (thinking it was a scam call). She said, I already did, and it’s real. After returning the table, the CEO asked me “What’s wrong?”, I relayed the information, and his first question was “Is this something that could put the company out of business?”, to which I replied, “I don’t know yet”. This was the beginning of a very long and hectic three months.

FBI On-site

True to their word, the FBI showed up the next morning, and we had all the necessary players around the table to ask questions and determine the scope of the situation. Before we could open our mouths the FBI proceeds to tell us that he’s not even a cyber crime agent, he’s a kidnapping/ransom agent. That he doesn’t know anything about cyber crime, and his job is to read us the information about the incident. So, he begins and reads a document that says the FBI had been monitoring some bad actors in eastern Europe and that our Company was 1 of 30 companies transmitting data to foreign IP addresses. After reading the statement, he gave us the specifics of which IP addresses were in scope. We start asking questions, to which he simply says “My job was to inform you of this activity, if you need more specifics, here is a card of a cyber crime agent that might help you”. We shake hands, the meeting is over.

Continue to Part 2

About the Author:

This blog was written by John Hensberger, Managing Partner of Black Bottle IT. Earlier in his career, John was also part of a company that experienced a cybersecurity breach. That experience fueled his passion for assisting other companies with their cybersecurity needs to mitigate their risk. As Technology Executive and Cybersecurity Advisor, John was recognized as the Pittsburgh CIO of the Year, 2014. Connect with John here.