The Govinfosecurity site recently reported on some very interesting scientific research that finds so-called “embedded training” is ineffective. Let’s quickly define “embedded”, they chose this word to describe what we in the industry call “point-of-failure” training where the employee in their normal work environment gets instant corrective feedback, often via a short training module after they fail a simulated phishing attack. The title of the article is a bit misleading and creates the false impression that awareness training in general does not work.

The headlines states: “Why Training Doesn’t Mitigate Phishing” – combined with a subheader that reads “embedded training is ineffective”. They are referring to research published in January 2014 done by a team from Vanderbilt University, MITRE and I3P Dartmouth College. There is an audio interview with Eric Johnson, a Vanderbilt University professor who is a co-author of the study.

A Bit More in Depth

Let’s dive a little deeper in this new MITRE/Vanderbilt study about embedded training and finish off showing that there are other approaches to security awareness training that definitely do work.

The study’s executive summary stated: “To explore the effectiveness of embedded training, researchers conducted a large-scale experiment that tracked employees’ reactions to spear phishing emails and immediate training activities. The results showed that many participants did not read the training, and clicked either all links or none across three trials.”

They ran three tests, with a 90-day period. They based their research on earlier studies to see if they could replicate their results. Their final sample included 1,359 participants and they were able to identify three groups: people that never clicked, people that almost always clicked, and the ones that only clicked sometimes.

Their so called treatment group landed on a page that showed them their error. Afterward researchers interviewed users and asked them what their reaction was after seeing this one-page corrective instruction. Many participants misinterpreted the training page when it popped up immediately after clicking a spear phish and feared it represented a further attempt to entrap them. Most all-clickers did remember a page popping up when they clicked, but almost none remembered specifics.

Most interviewees had taken only the company’s annual training, plus one or two courses required by their projects. Although they assumed that spear-phishing awareness was part of this standard training, few could remember it with any certainty. Almost all non-clickers considered annual company training ineffective; it repeated information that they already knew and instructed them to act in ways they already did.

“Unteachable Moments – Incorrigible Users?”

In their study, they tested the retention effects over approximately 90 days instead of 28 days, because they claim training every month in a corporate setting isn’t manageable. The study commented: “Company security might offer a false sense of confidence. Almost all interviewees believed that company firewall and information security programs prevented malicious emails from reaching them.”

Research Catch and Takeaway

They stated: “Making embedded training effective in a corporate setting is more difficult than earlier studies suggest. Our results indicate that immediate feedback and tailored framing don’t suffice to reduce click rates or increase reporting.” Another note was “This article reports on only one corporate study. The researchers conducted a second study, to be reported in the future, using MBA students with business experience. The study yielded similar results, reinforcing our conclusions. Nevertheless, for widespread validity, the study must be replicated at other institutions”. One of their hypotheses for the surprising results was: “Repetition might be required to achieve the training effects; participants might need repeated exposure to the training before their behavior is changed.”

Why Are We Not Surprised?

These results are no surprise here at KnowBe4. We have known for quite a while that this type of security awareness computer based training does not work all that well. There are methodologies of training end-users that are extremely effective though.

In this graph you see results on 372 organizations, 3600 simulated phishing tests each sending multiple emails, and just under 300,000 employees over 12 months. The numbers speak for themselves. The initial baseline phish-prone average percentage is just under 16 percent, meaning these employees clicked on our initial simulated phishing test. (Often these numbers were more than 20 percent.) Twelve months later, after having been trained and been sent regular simulated phishing attacks, this dropped to 1.28 percent showing a reduction in risky behavior by over 12X.

Security Awareness Training Results

The methodology used in this case was based on the following 5 steps:

  • Establish baseline Phish-prone percentage of all employees
  • Train all employees (On-demand web-based Security Awareness Training)
  • Test employees (Continued, Regular Phishing Security Tests)
  • Educate all employees (Ongoing Security Hints and Tips Emails)
  • Reporting and Tracking Results (Management Console)

The initial baseline percentage was often an unpleasant surprise for both IT and non-IT management, and cause for concern. The result though was support from the C-level on down for a sustained year-round security awareness program, with feedback on a regular basis, creating an effective security culture.

We think that the article’s title is needlessly confusing. Our position is that security awareness training programs done the right way can be extremely effective. Based on Kevin Mitnick’s 30+ year unique first-hand hacking experience, you now can train employees to better manage the urgent IT security problems of social engineering, spear-phishing and ransomware attacks.

Related Pages: PhishingSpear Phishing


Get the latest about social engineering

Subscribe to CyberheistNews