We have witnessed the in-source/out-source/in-source process for several Security Operations Centers (SOC) over the years. One of the most important lessons learned about in-sourcing comes from the staffing approach when standing up this capability.
Starting with junior resources
Several organizations believe they can save money by hiring L1’s, interns, or low level analysts at the start. While this does seem to make some sense because it allows you to ‘get in the game’ with a reduced up-front investment (right after you spent millions on putting all the pieces together!), this may not be as intuitive as one might think. In one specific case, a customer had hired 7 L1’s (juniors) working in a Security Operations Center without any senior guidance. The plan was to [eventually] hire a lead to manage the team and mature the program, but what may have eluded this particular team is that the lead they are seeking would be taking on 8 distinct responsibilities walking in the door – day 1. (YIKES! reference: “Giving up my life for my job!“)
The results at the end of this path are predictably chaotic and painful, rife with disappointment, failed objectives, and frustration. Placing the entire burden of: training juniors, implementing improvements, being the escalation point for incidents, and collaborating with other teams simply isn’t going to work, and few people will sign up for this (reference: “Why can’t I find anyone to take my awesome SOC lead job!?’)
Why it hurts…
We all hate to hear someone tell us “you have to crawl before you can walk or run”, but expectations need to at least be make sense, or you are going to hurt yourself. Let’s look at some lessons learned:
It typically takes 6 to 12 months for an L1s to gain enough experience to develop the mindset and independence to operate as a functional part of the SOC. This is especially true when there are no documented procedures. (But this totally never happens.. <.<)
Junior staff need an escalation point for their questions and challenges. If senior resources don’t have bandwidth due to conflicting priorities and workload, the process breaks down, morale falls, and the team loses impetus. Juniors may leave due to lack of development opportunity.
The senior resource can experience burn out with long hours, slow progress, frustrated leadership, and an inability to move things forward.
You may lose traction and progress when you get to your first SOC anniversary, have no strong result to show, and half the team leaves (reference: “Getting and keeping my resume up to date!”)
In this approach to staffing security operations, after 18 to 24 months a few of the junior resources will become skilled while others will continue to require more guidance, ultimately hurting team performance and limiting the overall program evolution. Again, this is because there is so little guidance and leadership direction in the beginning that the program foundation to build on simply isn’t there! At worst, someone may just call a ‘do-over’.
What seemed to make perfect sense sets you back, and winds up being more costly than starting with expensive senior resources.
Cohorts – a better approach?
Below is just an example approach to solving the described challenges . While your mileage may vary, the idea is to set expectations around a long term strategy that both fulfills the enterprises need and sets reasonable expectations for staff.
Example of what a successful 24 month timeline to in-source Security Operations looks like.
For example, an organization can consider the following as an approach (modified to meet their specific needs).
Bring in at least two senior resources at the start and two junior resources with some automation tooling. Ideally, <shameless_plug>you get help from people who have been down this road</shameless_plug>. Pair a senior resource with a junior resource, forming two teams
Have the teams share responsibility for assessing current state while managing day to day issues. The two junior resources can help while they learn.
Develop a road map of what is needed to make more junior resources successful and efficient when they start.
Implement iteratively based on priority and then begin on-boarding more resources.
Conduct retrospectives and feedback sessions with the junior resources during the whole process.
During this time frame, the initial two junior resources can provide guidance to new associates being on-boarded while the senior resources continue heavy lifting.
Key takeaways
The key here: set reasonable expectations with leadership for program success. An organization fairs far better if it retains senior resources versus burning them out and being forced to find new ones.
Begin with Critical Services, onboard in cohorts, then steadily mature the team and capabilities. This is how you accelerate progress versus investing with little result to show. Quick and frequent win stories make life better for everyone!
Indelible can help you rapidly develop a custom plan that fits your organization for maximum success.