Is a 100% Success Claim Realistic?

Guaranteed to give you a full head of hair…. guaranteed to make you lose weight… guaranteed to change your sex life…. These guarantees are often used in the marketing world to gain curiosity and to make the consumer take a peek at the offer.

Now come on, you don’t have to admit it … but guaranteed: most of us reading this have taken a second gander at one of these offers at least once.  I know I have.  However, even if you’ve fallen for this tactic in the past, you likely now realize that these claims are bogus or, at best, contain excessive amounts of fluff. And so we tend to cast a skeptical eye because we’ve finally accepted the fact that, when an offer seems too good to be true, it likely is.

What does this have to do with pentesting?

Unfortunately, in the world of security, we’ve begun to see a small trend in which companies are attempting to acquire customers by making a statement to the effect of, “we have a 100% success ratio” or “we are the only company that does X, Y and Z with a 100% success ratio.”

Many fellow pentesters reading this article will agree that a claim like this can be offensive, and also discrediting to the companies that feel they need to use this tactic to sell their services. Let’s examine this claim from two different angles; first, the perspective of the customer, and then the perspective of the security professional.

The Customer Angle

Imagine you are an IT or security professional working for a large company; you are tasked with finding the best partner to help you 1) develop a culture of security awareness, 2) protect your company from malicious attacks, and 3) act as your teammate and advocate. Can you imagine yourself in that boat?

Now, what is important to you as that person tasked to pick the very best and right partner for your organization?  Is it the fact that they claim to have a 100% success ratio?  Is it the fact that they claim to be the only ones on earth that do a certain thing?  Is it the fact that they claim to be the undefeated heavyweight champions of the universe in pentesting?

Probably not. As a matter of fact, if you even believe the ludicrous idea that anyone can truly say they have a 100% success ratio (more on that later), you wouldn’t want someone with whom there is no hope for you to ever succeed, would you?  Constant failure is no fun for you or your people.  Yet you aren’t looking for someone who will let you “win” just to stroke your ego.

How does one make an informed decision? The answer: you must ask the right questions and do your research. Here are some questions for you to consider when making this decision:

1.    How long have these folks been at it?
2.    What is their training background or education?
3.    What are they known for publicly?
4.    Do they have references or legitimate testimonials they can share?
5.    What have they written, spoken, or been interviewed about that gives you insight into their company and experience?
6.    Have they contributed anything to their field in order to better it?

This is not an exhaustive list, merely one that can help you to acquire the information necessary for making informed decisions.

The Pentester Angle

As a pentester, we have to ask ourselves, “ What is important to me as a pentester?”  Is it always “getting in,” “winning,” owning everything in my sight?  Of course, the hacker in me wants to “win” but the security consultant in me that cares about his customers takes a step back and repeats this motto; “I can think like the bad guy but always remember I am the good guy.” The end goal of any of our engagements is to make our customers more secure.

As the good guy, it’s important to remember there are real people, real feelings, and real emotions behind every company. Here’s a story to illustrate.  Five or six years ago, right around the anniversary of the 9/11 attacks in NYC, our CEO, Chris, had written a phish that was very popular in the wild, portraying itself as a charity for 9/11. The phish was strategically developed, well written, and ready to be sent to our client population. The email was first sent to the POC for approval, and they loved it. The phish was certain to get a high number of clicks and make people think. However, what we were not prepared for was the one person in the meeting who spoke up and said, “Just as an FYI, there are five people in the company who lost family or friends in those attacks.”

Did we know that using this phish would prove successful? Yes. But, it could also have the potential to be emotionally damaging for those five individuals. Would they truly learn a security lesson from this attack or would they remember how they were exploited using a terrible disaster? The goal of any security assessor and trainer is to ensure the organization benefits from the service. As professionals, our organization opted to do what was best for the training of this customer and its employees. We re-wrote the phish as a different charity. We then took a screenshot of the 9/11 email for the internal intranet training page built to educate employees. The email was used in a positive way to show employees what could be used against them without causing emotional damage that would lead to further security setbacks.

To this day, we still do business with that company and our relationship with them is strong.

But did we have 100% success?

The Truth Behind the Numbers

If you were to come and analyze Social-Engineer, LLC’s records and see what our success ratio is, you would be presented with 100% success; well, that is if we used marketing math.

Let’s say company X hires us to conduct a security assessment that involves breaking into four buildings, and the phishing and vishing of 500 people. Once completed, we have successfully entered three buildings, had a 60% click ratio on the phish, and influenced more than 80% of the vishing targets to give us their DOB and SSN. What is our percentage?

The way we do math is 75% + 60% + 80% = an average of 72%.  100% success would involve breaking into 4 buildings, receiving 500 clicks, and having every single employee blindly provide their date of birth and social security number to unverified callers. There are simply too many unknowns to make this claim. When companies claim 100% success, we implore you to take caution and look at the probabilities of such vast claims.

If a real attacker sent your 500 people a phish and 1 person clicked, they have compromised your company – that is success, right? Sure, but it is not 100% success. Not from a true security consultant’s point of view.

Is it really that important?

At the end of the day, you have to make decisions on who to work with.  But in my years of experience working on both sides of the fence, rarely was “I never fail at pwning my clients” a serious factor in my decision.

We’ll leave you with this question to meditate over.  If you are looking for a partner to help with your phishing, vishing, break-ins, social engineering consulting, or auditing – are you looking for someone who touts their amazing ability to always win, or someone who is ready to work with your organization to get a collaborative win?