I discuss security metrics with clients all the time. Sometimes it’s due to data shortages, sometimes it down to the resources needed to collect and report metrics. A lot of the time it’s because the metrics are currently in the red bracket and so the first line can sometimes feel uncomfortable reporting what looks like they are failing to maintain a secure environment. Most of the metrics discussions in smaller organisations are focussed at the operational level. In larger organisations there is a greater focus on the strategic level, the discussions are more about what should be presented to the Board. How the CISO effectively (and concisely) let the Board know the answer to the question “Are we secure?”. One very effective way to do this is to report on the return on investment the organisation is getting when investing in security – the organisation’s ‘Security ROI‘. In this article, we’ll discuss why it’s important to demonstrate security ROI and some simple ways a CISO can calculate a security ROI value for their programme. Because. if the CISO isn’t providing their Board with a security ROI figure, then it’s quite probable the organisation is spending too much on the wrong things.
What is Security ROI?
Security ROI is a risk-based figure reported in percentage terms. It shows the percentage return an organisation gets back for the money it apportions to cybersecurity. The metric can be wrapped up as one single overarching figure but can also be split down to a very granular level right down to an individual firewall or security analyst. Security ROI takes a long-term view and takes into consideration multiple events occurring throughout the life of investment in security resources. Security ROI can also be trended over time too.
But CISOs aren’t accountants!
A CISO is a strategic role. They don’t need to be an ACCA but they do need to have some financial accounting knowledge. A CISO must be able to demonstrate the organisation is managing its cybersecurity risk and that means one of the CISOs most important roles is to ensure their organisations are getting the best bang for the buck when it comes to delivering a risk-based cybersecurity programme. The CISO’s role is not to just throw money at cybersecurity tools but to make sensible investments in tools, human capital and intellectual property that minimise risk to the organisation but also doing so spending the minimum needed. No more than necessary.
Calculating Security ROI: Proving the negative
“You can’t prove a negative”
Ever heard a CISO talk about how they can’t show how successful their programme is because they can’t provide a number of how many attacks didn’t happen. Well, the thing is, most CISOs could easily provide data on how many attacks didn’t happen but they simply don’t report on those. Why not one may ask?
For those of us who have implemented Security Information and Event Management Systems (SIEM) we know from collating firewall, IDS/IPS, AV, Windows Event Logs…and the rest…that there is a white noise out there of constant probes against our networks. Trillions and trillions of potential security events are happening every day. Now I use the word probe as some of this activity is pretty benign. Just look at how much activity is recorded in your home router logs and you will see plenty of activity. It’s not because you are personally being targetted by Russian organised crime or a nation-state, it’s just everything is being scanned all the time. Take the Internet-Wide Scan Data Repository where if you had the time and inclination you could map the Internet. Then there are kids in bedrooms practising things they have learned on YouTube (Scan the Internet anyone?). Take it up a level and you have IT developers building new tools and security researchers trying to identify vulnerable systems. Because of this white noise, many organisations simply ignore what has been blocked by their security tools or don’t have the sophistication to filter out the noise from the genuine directed attacks. But, by not demonstrating what the security tools are keeping out, information security teams are missing a massive opportunity to show how effective their security tools are at protecting their organisation. These CISOs are failing to show that their organisations have a good (or bad) security ROI. By not knowing if you have a good security ROI, you have no accurate way of knowing cybersecurity risk is being managed at the most optimal level.
Ever wondered why other people get their projects funded but not you?
If you don’t show how good an investment you are, why would anyone invest in you in the future?
Have you ever been in the position where you have spent ages producing the business case for a new security tool only to be told to go away and provide some more information about what other organisations are doing? Perhaps you have been asked to produce a paper for the next Board meeting (in three months time) highlighting the risks of not proceeding with the purchase. These exercises in sophisticated procrastination are common. This is because senior management doesn’t like paying for things they see as pure costs but they have to be seen to be doing something about managing risk. They know they have to spend some money but it is often a bitter pill to swallow when they know that investing in a SIEM will mean they don’t have the CapEx/OpEx to spend on new revenue-generating products. If a decision to spend money can be avoided, a way will be found. Part of the reason for the management wanting to avoid the spend is the return on investment of such tools are not really understood. True revenue-generating business units understand that when getting budget sign-off for their new system they have to show in dollar terms how much money they expect a new venture will bring in and thus why the investment in the supporting tools is so important to achieving that goal.
“They show in $dollar terms why they need that new system, resourc or application – do you?”
Now it’s pretty easy for a CISO to say they don’t generate revenue so it’s not a level playing field”. Or:
“How on earth can a CISO demonstrate security ROI on a firewall, it’s just a cost of doing business?”
Sure, you could ask such a question or, what you should really be asking is:
“How much would it cost the business if that firewall was stripped out of the network?”
What would happen? When you flip it around to assessing the implications of removing a piece of security infrastructure, it’s pretty easy to see why it’s there in the first place. What is the cost of extra unwanted network traffic? What is the cost of system downtime? What is the cost of a data theft? All these costs can be calculated pretty easily. Other departments calculate their metrics in financial terms, why not the information security functions? Most organisations know how much they are paying for their current headcount. It’s pretty easy to work out how much system downtime will cost in terms of lost revenue or lost opportunity. There are even estimated data breach figures on a per customer record basis. Do a bit of research and you can find data such as customer attrition rates in the company reports of organisations who have had data breaches. TalkTalk, for example, had this to say in their 2016 annual report:
“H2 revenues were directly impacted by the cyber attack which resulted in a spike in churn and an extended period over which we were unable to trade from our online channels.”
There is also a clear longer-term increase in customer churn demonstrated in the following chart taken from TalkTalk’s 2019 annual report:
As you can see there is a spike in customers leaving around the period of the cyber attack. Bear in mind that for a telecom provider, customers are often in fixed contracts and may have had to wait until the end of their contracts in order to leave. One could also consider that this is a chart showing customers leaving. There is also the reputational damage which raises the cost of converting new prospects into paying customers. The business knows these metrics are important as this data is going into the annual report. It should, therefore, be pretty obvious that CISOs need to be demonstrating security ROI by linking effectiveness to financial metrics.
I don’t have enough data to make an accurate security ROI calculation
A good risk manager knows models are not 100% accurate. I’ve mentioned in other articles that models don’t have to be perfect representations of the real world, they just need to be good enough to have a reasonable level of accuracy within confidence bounds. The same can be applied to calculating security ROI. What you need to do is work with what you have and as more data becomes available, improve the model. Don’t wait until you have perfect data but instead improve your confidence levels as you add more data. Here are a couple of very simplified worked examples with made-up figures:
Changing a Product
Let’s say a business with a turnover of $1,000,000 is thinking of changing its AV product. The current AV product costs $1,000 to implement and $1,000 annually to run (licences, infrastructure, resources etc). The cost of ownership over 3 years (36months) is $4,000. But that’s not the whole story. Industry ‘statistics‘ (I don’t actually believe this but I needed a figure) suggest a ransomware attack (downtime and restoration) lasts on average 7.3 days and so could cost this business $20,000 per infection. A SOC Analyst checks the current AV logs and reports that the current AV product has picked up and quarantined on average 1 piece of ransomware per month which would have cost $20,000 x 36 months = $720,000 over three years in downtime and restoration costs. The AV solution is not perfect and does let through 2 infections per year (costing 2 x 3 x $20,000 = $120,000), In total there are 14 (12 + 2) ransomware events per year. So a security adjusted Total Cost of Ownership (TCO) is $4,000 + $120,000 = $124,000. The security ROI of the current AV tool is, therefore, $760,000 / $124,000 = 610% for a three year investment. That means the organisation gets 6.1 times back what it invested in the AV tool in terms of revenue which could have been lost if the AV product was not installed.
The business is not happy with the 2 incidents of downtime per year and has now lowered its risk appetite for disruption. Senior management has asked the CISO to make some recommendations. As it happens, some amazing up and coming AV vendor comes along with the next big thing in AV. This new solution costs $20,000 to set up and then $5,000 per year to run. The future cost of AV is now $45,000 over three years (up 11,250% from the existing AV solution). If you went to the CFO with this they would probably think you’ve gone mad for even suggesting such a massive increase in spending.
However…
It just so happens one of your similar-sized industry peers have been using this wonder product for the last 6months. As luck would have it they have exactly the same infrastructure as you with the exception of this tool (handy eh!). Their logs show this solution only lets in 1 infection per year instead of 2. So the TCO is $20,000 + $15,000 + (3 incidents at $20,000) = $95,000. The potential security ROI is now $760,000 / $95,000 = 800% for a three year investment. By changing AV tools the CISO could potentially realise a 190% increase in ROI from the existing setup. You now have a more compelling story to tell the CFO.
Defence in Depth
The AV example is, of course, highly simplified to get the basic premise across. Security tools do not work in isolation and calculating security ROI is a lot more complicated. Security tools work as part of a layered approach to managing risk. With a little bit of thought, the TCO and ROI methodology described above can still work though when considering a bigger integrated security apparatus consisting of multiple tools and resources. You could use the same methodology to see if instead of changing AV products, could implementing a DNS-based web filtering solution in parallel with the current AV product prevent the remaining 2 ransomware infections? Would this combined approach offer a better overall security ROI?
Is awareness training on its own value for money?
Take another example, what is the TCO of employee awareness? What is the security ROI for your training and awareness programme? Don’t have the data? Get an external organisation to run a phishing exercise and for every one of your staff that clicks on the phishing link treat that click as a hypothetical zero-day ransomware infection. You now have some data. Let’s work through it. Say the same organisation in the previous example has 1,000 employees and spends $1 per user, per month, on annual Computer-Based Training ($12,000 per year). They pay $5,000 for an external company to do a phishing exercise (cost so far is $17,000). 40% of users click on the link in the phishing exercise email. That’s 250 potential ransomware infections (with a potential cost of 250 x $20,000 = $5million). That figure is 5 times the annual turnover of the company so it’s clear if further investment is not made, multiple outbreaks could easily put the company out of business. Something must be done, but what? This initial exercise shows the organisation’s CISO that the CBT has a materially negative security ROI. The shrewd CISO, however, can use this evidence to change the approach. Instead of generic CBT the savvy CISO then combines monthly phishing exercises with follow-on training targetted at those most vulnerable There is a lot more effort involved and so the cost is now 12 x $5,000 = $60,000 but the click rate drops to a staggering 2 a year ($40,000). Security TCO is now $100,000. Nominal Security ROI is now $5,000,000 / $100,000 = 5,000% but as current total revenue is capped at real-life values a more realistic security ROI is $1,000,000 / $100,000 = 1,000%.
Now obviously, this is another example of where a layered approach to security would show that spending £100k on phishing exercises may be excessive and that a combination of approaches would yield a sweet spot in terms of security ROI at a lower price point. But without going through the exercise it would be otherwise very difficult to tell whether there was a net positive security ROI for further training without having to go through a few ransomware infections first..sure we have an instinct what will happen but wouldn’t it be better for everyone if we could use an evidence-based approach when describing the risks associated with our businesses.
What’s a good security ROI?
The short answer – it depends. Ultimately it’s down to the CISO and the CFO to work together and come up with a figure appropriate to the organisation’s risk appetite. Whatever the figure agreed, security ROI must be a metric reported to the Board on a periodic basis. It’s a metric the first line of defence should be held to. In mature organisations, it could be a very good risk-based KPI for each head of business – not just the back office functions owned by the CIO or COO. The CISO’s role would be to make recommendations as to how security ROI could be improved and to validate the methodology used to calculate security ROI has been followed correctly.
Establishing a framework to calculate and monitor security ROI can be daunting. Fox Red Risk can help. If you want to ensure you’re organisation is getting the best bang for its buck when it comes to cybersecurity investments, get in touch.
About the Author:
About Fox Red Risk:
Fox Red Risk is a boutique data protection and cybersecurity consultancy and Managed Security Service Provider which, amongst other things, helps client organisations with implementing control frameworks for resilience, data protection and information security risk management. Call us on 020 8242 6047 or contact us via the website to discuss your needs.
22301 22301:2019 article 28 awareness bcms BIA business continuity calculating risk change management ciso controller cybersecurity data breach data privacy Data Protection data protection by design data protection officer data protection service Data Subject Access Request DPO DSAR GDPR incident management information security leadership monitoring operational resilience Outsourced DPO Privacy processor risk risk appetite risk management ROI security security as a service small business strategic strategy Subject Access Request tools training transparency vciso virtual ciso
