Is your Managed SOC starting to smell a bit fruity? Here's what to do if it is

Remember a couple of years ago (when life was so very different). Remember reaching that point in your security maturity journey where you needed a way of detecting security events without the help of the BBC News letting you know? Remember looking at all those complex SIEM solutions? Remember deciding the time and effort involved to run a 9-person 24-7-365 team of SOC analysts, the cost of tin and, the never-ending storage requirements to collect all those logs and packets, needed to run a Security Operations Centre (SOC) in-house, was just too much? Remember deciding it would be a lot simpler to let a third-party vendor run it for you and so a Managed SOC solution was procured. A few years have since elapsed and you’re beginning to wonder whether you’re getting any value for all that money and, more importantly, whether the SOC is even fit-for-purpose. If you’re facing this situation, or if you’re just embarking on your journey into the wonderful world of outsourced security operations, what can you do? Read on for some top tips to avoid the common pitfalls organisations start facing around years 2-5 of their outsourced SOC adventure…

What is a Managed SOC?

It’s probably a good idea to explain what a Security Operations Centre is. For the purposes of this discussion a SOC can be defined as:

“A dedicated site where enterprise information systems (web sites, applications, databases, data centers and servers, networks, desktops and other endpoints) are monitored, assessed, and defended.“

A Managed Security Operations Centre (MSOC) or Outsourced SOC is pretty much as described. Rather than implementing and running the as-described Security Operations Centre in-house including the implementation and operation of a Security Information and Event Management (SIEM) solution, you pay a third-party vendor to manage it for you. At a very basic level, the organisation provides the logs into a central logging platform and then the logs are shipped into a SIEM. From there, machine learning of various forms occur and alerts are then triggered, and dashboards are updated in real time. Alerts and dashboards can be derived from out-of-the box use cases developed by the SIEM vendor or can be customised by the end user. Depending on the MSOC model, the vendor may keep the logging platform and/or SIEM fed and watered whilst hosted in your environment. The vendor may also host the logging platform and/or SIEM remotely (I hate saying “in the cloud”). In other cases it may simply operate the SIEM from the front end whilst your organisation does all the administration and hosting of the logging platform and SIEM internally. The MSOC may also provide enhancement to the core SIEM in the form of third-party alerting tools – again these tools can be hosted internally or with the vendor. Typically these third-party tools are there to make the vendor’s life easier by signposting specific activity, so the vendor doesn’t have to develop their own detection and alert rules. Wherever the data and systems are hosted, the MSOC should always provide the monitoring and reporting of security events. The MSOC vendor do this by employing the SOC Analysts on your behalf and are responsible (in theory anyway) for their training and development. Running a SOC is a complex affair, the above is a very basic overview!

What about the SIEM, surely that’s also important?

You may notice I haven’t made much mention about which SIEM or logging solutions are out there. Suffice to say, the solution is less important for this discussion compared to the service wrapper. BUT it is still important. When it comes to their underlying platform, MSOC vendors typically go down one of two routes. Some opt for commercial enterprise SIEM solutions, provided by a software vendor. This option is often preferred by those vendors who don’t have a mature in-house development capability. Others opt for Open-Source solutions. This latter option is often preferred by a technically mature vendor and is typically (but not always) supported by in-house level three engineering expertise. Why do these two options matters? Basically, because it affects the speed of delivery for any custom request. In the first scenario, if you want a custom use case developing, the request goes from you, to the MSOC vendor, and then off out to the SIEM vendor’s development team. As a result the process is slow and could take months before something is implemented in the product – if at all. In the latter case, the request is typically fulfilled by the MSOC development team and, as the work is prioritised amongst a smaller user group, is typically delivered in a shorter timeframe. The latter often has the benefit of significant additional support from the open-source development community whereas the former is solely reliant on the SIEM vendor’s internal team – and they’re pretty busy bees! It’s definitely worth asking the question about customisation…even better, give them a custom use case to demonstrate during the pre-sales process.

So, back to main discussion. What should be considered when procuring a MSOC?

Caveat Emptor (Buyer Beware)!

The most important thing to consider when procuring an MSOC service is “Will the service procured be fit for purpose?” The second question likely to follow is “What does good look like?” You may not know how to answer these questions. If this is the case, talk to your peers who have already been on the journey, ask them what has worked and what has not gone quite so smoothly. It’s also worth throwing a bit of money at the procurement process and get in a security consultant who can help support you in your SOC procurement. If you do get someone in, make sure they are vendor agnostic (Like us at Fox Red Risk) so you know the consultant will be looking out for your needs, without one eye on how much additional commission they will be getting from the vendor they “choose”.

If you are going to go it alone, do not be vendor-led! Don’t let the vendor tell you what you want. List your own requirements, perhaps in the form of an RFP, then get SOC vendors to show you how they will deliver on those requirements. When I say “show”, I mean a live demonstration of the end-to-end service supported by a laboratory-style environment. Do not let them put anything in your infrastructure until they have proven the basics in their own laboratory environment first. If they don’t have a laboratory environment, that is a major red flag. Equally, don’t rely on pre-recorded “perfected” demos either, SOC is real-time so the demos must be real-time too. Do not be swayed by PowerPoint sales fluff about how they are growing exponentially, are on a quadrant or are working with a number of your peers. The vendor may very well be working with your peers but that doesn’t mean they are getting good service – or you will get the same service! They may also be growing but what does that mean for the service you will get once the sales guys have moved on to their next prospect. Make sure you have their technical people on the call. Some salespeople don’t like this, but you must insist. The technical guys are typically more candid (read honest) about what is actually possible, and you need to know what is and isn’t possible before signing on the dotted line.

A SOC that works for you…not that costs you.

Not all SOC services are created equal. Some outsourced SOCs don’t actually know how their SIEM solution works but use it as a hub for alerts created by middleware. The MSOC will often be a service wrapper around the licences to a number of middleware alerting tools and then, in addition to the cost of the underlying infrastructure and storage, you pay for a share of some SOC Analyst’s time to tell you those alerts have triggered. Such vendors often have little in-house engineering capability and, as mentioned earlier often rely on the product vendors for any custom work – which can take an age to get delivered! You will know you’re dealing with this type of vendor because you will not have been asked anything about your infrastructure, what is important to you, or what activity you’re interested in detecting. They are, in essence, little more than an enhanced software reseller.

Some outsourced SOCs do rely on a SIEM solution at it was intended to be used. They have some good first line analysts too. The first year goes fine, as the service beds in, but the service is never tuned, and data keeps building up – and so too does the cost of the service. Requests for Change come at a premium and, as change and capacity have not been factored into your budgets for years 2 and beyond, the system is now draining your budget and taking money away from other parts of your security programme. You will know you’re dealing with this type of vendor because there is little or no planning of future capacity at the outset. They may also offer discount rates for year one or base future year costs based on static infrastructure (i.e. zero change) – and we all know IT doesn’t stay the same for long!

Some outsourced SOCs have a product that they think works and don’t want to customise to the needs of the client. They feel they are the experts, and the client should simply treat the service as a black box, “Don’t worry” they say, “we’ll let you know when something happens”. They won’t provide evidence that the system is working, often citing a need to keep data from other clients confidential. This should of course ring alarm bells in regard to where your data may end up but more importantly means your specific needs may never be met.

Some outsourced SOCs are on a journey themselves. They developed a Minimum Viable Product (MVP) but then quickly reached the limits in its capability or capacity to serve an ever-increasing customer base. They’re so busy growing, the underlying solution is creaking under the weight. When you first signed up, you didn’t notice any of this because you had the onboarding “A-Team” and were treated like royalty. As soon as you signed off that initial project into Business as Usual (BAU), the wheels fall off. Suddenly responses and requests seem to fall into a black hole.

In all Outsourced SOCs, there can be further challenges in the service wrapper itself. Challenges can occur when sales’ teams make unrealistic commitments. Pre-sales write Statements of Work that can’t be delivered by the Project Team. Project Teams kick issues over the fence into the Operations Team. SOC Analysts mishandle customer change requests as incidents. The list goes on. The primary SOC could be doing a great job in principle but all these service issues add up. The result is a sour taste in the client’s mouth. You’re now at the stage of weighing up whether there was ever value for money in outsourcing the service and are considering putting in notice with the vendor. But it doesn’t need to get to this stage!

So, lots of challenges to overcome. They are not insurmountable by any means, but it does mean you need to know what you’re doing when you procure an Outsourced SOC service and you need to know what questions to ask. We’ll cover those first and then move on to what to do if you’re at the end of your tether.

What do I want to want to protect?

If you don’t know what is important in your organisation, how on earth is the MSOC provider going to know either! That said, a good MSOC vendor should ask this up front. If an MSOC vendor doesn’t ask you what you want to protect then how do they know what is critical to your business? How do you know if your critical information system assets will be in scope of the service? If you are procuring an MSOC service, it is highly likely you already have a mechanism to assign a criticality rating to you IT assets. If you don’t already do this and would like some help, get in touch. At a basic level, one simple way to identify what is critical is to look at where you conduct penetration testing. Penetration testing is often initiated earlier in an organisation’s security programme than the establishment of a SOC and so work will have been done during this programme to identify what infrastructure needs testing. Therefore, if there is a reason you conduct penetration testing on those assets, equally you may want to ensure attempts to compromise those assets – in between tests – are quickly detected too. Basically, try and get assurance from the MSOC vendor that the activity from a pen test will cause the MSOC dashboards and alerts to light up like a Christmas tree!

What are the use cases I want the SOC to detect?

There are Use Cases and marketing “use cases”. The latter makes it look like a lot is happening but as if often the case, the devil is in the detail. It’s important when procuring an MSOC service, once critical infrastructure has been identified, that organisations then move on to determining what activity they want to detect. You don’t have to come up with these use cases yourself either. There are models such as the MITRE ATT&CK framework that can aid an organisation in building initial monitoring capability, gradually maturing that capability over time. Remember you don’t have to do it all at once as Katie Nickels very eloquently explains here (YouTube – “Putting MITRE ATT&CK™ into Action with What You Have, Where You Are”).

Once agreed, Use Cases must be formally documented to an appropriate level of detail. If you want a good practice example of what the appropriate level of detail should look like, then you can’t go too far wrong with the example* on pages 133-143 of Blue Team Handbook: SOC, SIEM, and Threat Hunting Use Cases.

Why is it important to document each use case to this level – it removes ambiguity. For example. Your MSOC provider states in their marketing material that they “monitor for unauthorised login attempts”. “This is what we need”, you think to yourself. “Anyone attempting an unauthorised login to our cloud portal is something we want to know about immediately – sign us up!” With a fully documented use case, however, you would see that the logs to support this use case are collected solely from Active Directory. Your cloud portal’s access management system, however, is completely independent from Active Directory. Whilst you’re basking in a false sense of security that your cloud portal is monitored, the MSOC vendor can fall back on the ambiguity of “That’s not what we meant”. Don’t expose yourself to unnecessary risk by failing to fully document your Use Cases.

What data sources do I need to collect in order to achieve the agreed use cases?

Before the Use Cases can be turned into a workable monitoring, detection, alerting and reporting solutions, the underlying data that support these activities must be collected. A good MSOC vendor will work with you to map your critical assets to your stated use cases and tell you what log sources would provide the level of detail required and the frequency of monitoring. Following on from this mapping activity should be an analysis of the minimum fields of data required from each log source, and any supporting enrichment data source (e.g. a Threat Intelligence feed) to make the Use Case viable. This information must be included in the Use Case documentation.

What are the projected service costs for the services over 5 years?

The next stage is to project the costs for the service over double the initial engagement. If you’re signing a two-year contract, then project the cost of the MSOC service over four years. A good rule of thumb is to look at your organisations current and projected rate of expansion – and then double it. As a backstop, model for the MSOC service costs to triple each year. It’s not to say costs will triple each year but it will definitely serve as a warning of what the MSOC could cost if the relationship is not managed effectively.

Once you have this information, you’re now in a position to document an achievable Statement of Work (SoW).

Is the Statement of Work achievable?

For those who don’t know your MSOC implementation is most likely to be delivered under contractual terms. There will be a Managed Services Agreement (MSA) and underneath this the nuts and bolts of delivery will be included in a Statement of Work. The Statement of Work is an important document because it explicitly states what will be delivered and what will not – the content of this document will either make or break the MSOC service. A poor SoW, filled with ambiguity or woolly undeliverable statements, will only lead to pain on both sides of the engagement. Some pre-salespeople think they are being clever when they make things “open to interpretation” because it means they have contractual wiggle room if an aspect of the service can’t be delivered or isn’t delivered to the customer’s expectation. The thing is, whilst they may be able to offload the engagement into operations, the operations team now have to deal with all the customer’s complaints. An MSOC vendor who doesn’t keep an eye on this can often find it is costing a lot of money, both in terms of diverting operational resources away from BAU but also the cost of acquiring new customers to make up for those they are haemorrhaging at the end of their initial term.

Make sure the SoW is clear, unambiguous and that delivery can be evidenced. A good Statement of Work should include:

Scope: Clear and unambiguous statements that describe what will be included and what will not be included within the MSOC service.
Deliverables: Clear and unambiguous statements as to what will be delivered, when and how delivery will be evidenced. If the MSOC is to deliver a specific set of use cases, these should be individually listed (but ideally in a signposted Schedule for ease)
Process: Clear and unambiguous processes for each aspect of the MSOC service delivery. A process should be available for incident escalation, but also for Requests for Change (RFCs) and Administrative Tasks.
Acceptability: Clear and unambiguous statements detailing the acceptance criteria for each stage of delivery, up to and including business as usual. If a set Use Cases are to be delivered then the format of those Use Cases must be agreed in the SoW.
Service Level Agreements: Clear unambiguous statements regarding the minimal level of service to be delivered before penalties are incurred. Vendors often don’t like committing to SLAs because where they have failed to deliver it means they will have to give money back to the customer. Wherever possible though, it’s in both sides’ interests to have SLAs for Incidents, Administrative Requests and Change. These SLAs should then form part of a periodic service report.
Timelines: Clear and unambiguous details of timelines for each phase of delivery. A new phase must not commence until the Customer has accepted the prior Phase. Once in BAU, there should be timelines for items such a Service Management Meetings too.
Requirements: Clear and unambiguous details of what is required to deliver the MSOC service including who is responsible for delivering on each requirement. If a specific use case requires logs from a specific device, in a specific format, this should be documented.
Costs Estimate: A list of all the associated costs related to the service. There should be a breakdown of costs and pricing to aid in future planning. No one wants surprises on this front! For example, you expand your Windows Server Fleet, what would be the cost of adding a new server (of the same configuration) into the scope of the Service? What would be the cost of removing a Server that has been decommissioned? These are just a few of the questions you would want answered in the SoW.
Payment Schedule: Pretty much as described. When will fees be due and where service has not been delivered how will refunds be handled. The payment schedule, at least for the initial delivery, should be linked to your phased timeline. If something has not been delivered, payment should be withheld until deliverables have been properly accepted. Nothing focusses a vendor’s mind like a delivery dependent payment schedule.
Termination Clauses: Whilst the MSA will include termination clauses covering the MSOC service as a whole, there should also be a detailed set of termination clauses for each sub-component of the service. Software licences for example could be negotiated on an annual basis within a three-year contract to cover expansion or contraction of service need. It is therefore important to make sure the SoW gives both the vendor and the customer the flexibility without having to create a completely new side contract.

Holding the Vendor to the SoW

Having a wonderfully crafted SoW is great but if a customer doesn’t hold the vendor to it then it is as good as useless. As mentioned in the last section, ensure there is an agreed phased timeline for delivery linked to a payment schedule. Ensure meetings are scheduled in to review what has been delivered and whether it meets the acceptance criteria. If it doesn’t, let them know payment won’t be released until the acceptance criteria has been met. Be wary of agreeing to requests for part payment for partially delivered items. This can often be used as a way to keep the project going without delivering on key components of the service – and ultimately it is you as the customer who will suffer for any gaps you accept.

Quite a few things to think about. If applied properly, they should get you a good initial MSOC service to then build upon. But what can you do if you have perhaps missed a few of the steps above – or missed them all completely? Can your MSOC service be salvaged?

Ok, but I am already in the middle of a horrible engagement? What can I do?

All of those above steps are great you say…IF…we hadn’t already dipped our toes in the MSOC world. What do we do, those of us in a position where we are in the middle of an engagement, are shelling out a tonne of cash and getting very little service in return?!

Feeling trapped in an expensive MSOC engagement can be an awful experience. It is oft discussed by CISOs who must fight for every penny of their budget. Knowing how hard they fought, accepting that the service is not fit-for-purpose can be physically painful – gut wrenching. It adds more stress to an already stressful job. If you’re in such a position what can you do? Obviously, the first step is to admit there is a problem….then…

Audit the Service

The very first thing you can do is audit the current service. If you can’t do this yourself get someone in like Fox Red Risk to conduct the audit. When commissioning audit, adopt a two-stage process. The first stage should focus on whether the agreed SoW has been delivered and the second stage of the audit should benchmark your MSOC service against current good practice. It should be understood that the SoW for the audit should be equally as robust as the SoW for the MSOC service itself!

Raise service issues at the next service management meeting

Once armed with a list of independently assessed issues, the next step is to talk to the MSOC vendor and establish a plan to remediate the issues identified. A good vendor will want to cooperate with the audit and work with you to get the issues identified fixed. Now, if the SOC vendor is meeting the requirements of the contract but the service is not actually fit-for-purpose, that’s not the end of the story. Whilst it may mean some additional one-off investment, funding remediation work to get the service to where it needs to be, can often be more palatable than paying for a white elephant for X number of years until the term expires.

Prepare your exit plan and give notice

In most cases, the preference is to fix what is in place, but if the vendor is not cooperative or able to meet your MSOC needs then it may be time to think about consciously uncoupling. Migrating an MSOC to another provider or in-house is not something you can do by simply flicking a switch – it will take some planning. Notwithstanding that, the contract probably has a sizeable notice period, typically six months or more. Some contracts have clauses that mean the contract will automatically renew if notice is not given. If you’re in a long-term contract with a bad service provider – and that contract automatically renews – it’s worth checking with your legal team to understand your options for exiting before you’re saddled with another X years of pain!

That six months’ notice period is not necessarily a bad thing. Organisations will most likely take that time to identify and contract with another MSOC provider. Use the time wisely. It will go very quickly. Because if you don’t get your ducks in a row before your current contract ends, you could find yourself with no SOC at all!

As always, if you need support with any of the issues discussed in this article, just drop us a line here at Fox Red Risk, we can help!

Is your Managed SOC starting to smell a bit fruity? Here’s what to do if it is

Leave a Reply Cancel reply

Fox Red Risk Solutions Ltd