A framework for competence development and assessment in hybrid cybersecurity exercises

Computers & Security 88 (2020) 101607

Contents lists available at ScienceDirect

Computers & Security journal homepage: www.elsevier.com/locate/cose

A framework for competence development and assessment in hybrid cybersecurity exercises Agne˙ Brilingaite˙ a, Linas Bukauskas a,∗, Aušrius Juozapavicˇ ius b a b

Institute of Computer Science, Vilnius University, Vilnius, Lithuania General Jonas Žemaitis Military Academy of Lithuania, Vilnius, Lithuania

a r t i c l e

i n f o

Article history: Received 13 June 2019 Revised 3 September 2019 Accepted 5 September 2019 Available online 6 September 2019 Keywords: Cybersecurity skills Cyber defence exercises Competence assessment Hybrid exercises Competence development framework Cybersecurity trainer’s questionnaire

a b s t r a c t Rising numbers and sophistication of security threats in the digital domain cause an increase in the demand for skilled cybersecurity professionals. In response, cybersecurity exercises, and in particular—cyber defence exercises (CDX) are becoming ever more popular. They provide a training platform to simulate real-life situations. CDX are significant events involving months of preparation, and previous studies show a lack of objective evidence of their relevance regarding the learning impact. Skills of exercise participants are usually different and vary from tech-savvy to beginners. Also, trainees are diverse when considering their background, current work profile (position and institution), and experience. Assessment of their competencies is essential to ensure quality in training. The complexity and multi-dimensionality of the usual CDX make it challenging. Additionally, the costly event usually focuses on just a subset of participants, and non-technical members of an organisation are not included. The goal of our research is to provide a proper methodology to optimise the exercises so that every team and each participant, including a non-technical trainee, are adequately evaluated and trained using the allocated resources most effectively. This paper presents a framework to aid in the development and assessment of cybersecurity competences of all teams during hybrid CDX. The framework aims towards raised cybersecurity awareness—a state when every user of digital technologies understands the associated risks. The framework consists of a sequence of steps including stages of formative assessment, team construction, determination of objectives for different types of teams, and the exercise flow. It complements standard methodologies for cybersecurity training programs. The framework was developed based on data collected using questionnaires, interviews, and direct observation in a case study carried out during international cybersecurity exercises. The framework would help organise hybrid exercises for a diverse community of trainees, including non-technical members of an organisation. © 2019 Elsevier Ltd. All rights reserved.

1. Introduction An increasing number of cybersecurity incidents and threat actors affects businesses all over the world (ENISA, 2019; Morgan, 2019). Capabilities to protect against or discourage cyberattacks are strongly related to readiness and competence of professionals at the scene as well as cyber-literacy and awareness of the general population. Cybersecurity exercises are used to alleviate the situation and train specialists. Most often, the exercises are organised for a dedicated and homogeneous group of professionals, and traditional methods to evaluate participants and compare their achievements can be used. A narrow target audience allows the organisers to streamline exercise objectives. However, this strategy ∗

Corresponding author. E-mail address: [email protected] (L. Bukauskas).

https://doi.org/10.1016/j.cose.2019.101607 0167-4048/© 2019 Elsevier Ltd. All rights reserved.

fails to ensure cybersecurity awareness and readiness at all levels of user involvement in the digital world. Cybersecurity assurance in an organisation relies on both hard and soft skills of all its employees. Technological and socioorganisational solutions and skills should be combined as a human is a critical player in each organisation (White et al., 2004). Formal cybersecurity education systems in high-tech fields tend to stamp a cliché on peoples’ behaviour and prioritise technical skills. Soft skills usually remain neglected, although they may be essential in achieving a successful cybersecurity career (Joint Task Force on Cybersecurity Education, 2017). In real life, technical and security staff is surrounded by people having different levels of education in diverse fields. General competencies for cross-disciplinary communication are necessary. Also, technical solutions have to be communicated to different organisations at different levels of hierarchy and different levels of complexity. Aoyama et al. (2017) map

2

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

types of exercises to the degree of organisation cybersecurity preparedness to reduce misuse of resources. Cyber Defence Exercises (CDX) is the most common and expensive approach to train, test, and verify the professional skills of organisation workforce at the highest preparedness tier. European Network and Information Security Agency (ENISA) recommends to include the CDX as a part of the national cybersecurity strategy (ENISA, 2016). Exercises on a regular basis (ENISA, 2018) should test standard operating procedures of a state. However, specialist-oriented exercises are not enough. Cybersecurity awareness and skills of a larger audience can be improved if decision-makers and the public, in general, are included (Ogee et al., 2015), because IT and cybersecurity skills will be required in all future jobs. A growing number of technical cybersecurity exercises and various hackathons attract many technically skilled participants. The events also contribute to increased cybersecurity awareness and welcome beginners to the field. Due to the nature of CDX exercises, participants with different background are involved. They form opposing teams with dedicated responsibilities in complex simulated real-world situations. We raise the research question whether CDX is an appropriate tool to develop and assess cybersecurity-related competencies of all participants and if the concept of CDX can be improved to address the learning needs of all trainees including non-technical persons. Based on our observations and findings in a case study, we developed an improved organisational framework of hybrid CDX to maximise the learning effectiveness. The framework complements the usual CDX life cycle and adds new phases to facilitate competence assessment. We present the steps to apply the framework to quantify the training result of all participating training entities. The framework encourages CDX organisers to involve non-IT specialists to increase the resilience of an organisation against cyber threats. The structure of the paper is as follows. Section 2 elucidates CDX-related challenges found by other researchers. Section 3 describes the methods we used to analyse the learning experience of participants during international CDX. We present our findings in Section 4 followed by a discussion in Section 5. We finalise our paper with conclusions and possible future directions in Sections 6 and 7, respectively. 2. Related work We group related work into several categories based on challenges related to competence development and assessment in cybersecurity. A lack of cybersecurity workforce is identified globally, and regional initiatives try to address the problem. In Europe, ENISA leads information sharing and development of guidelines, conventional approaches and procedures related to cybersecurity. In the United States, NIST developed NICE cybersecurity education framework (Newhouse et al., 2017). The widely accepted framework specifies knowledge, skills, and abilities to perform specific work roles in cybersecurity. Paulsen et al. (2012) distinguish cybersecurity awareness as one of the vital NICE components. An increased cybersecurity awareness can be achieved through high(-er) education and training of professionals. Parrish et al. (2018) provide the framework for the cybersecurity integration into the existing computing programmes defined in the ACM Computing Curricula series. They present cybersecurity as a meta-discipline that goes beyond the computing and engineering education and points out that cybersecurity is a cross-domain issue. Therefore, cybersecurity-related components are defined, and models of exposure to cybersecurity for all students regardless of their study field are provided.

Education. Cyber attacks should be treated as a precondition to all significant events (Ohta et al., 2018), but despite advances in automating cyber attack detection, the primary defence element is a trained human cyber defence specialist. Joint Task Force on Cybersecurity Education (2017) emphasises the urging necessity and promotion of education in cybersecurity. Many student-oriented competitions and exercises are organised for cybersecurity training and education. On the one hand, competitions are used as sources to recruit talented people into the cyber warfare forces (Andress and Winterfeld, 2014). On the other hand, CDX are being integrated into formal education as a part of courses of cybersecurity study programmes (Mauer et al., 2012). Hoffman et al. (2005) list four types of cybersecurity exercises organised for students in educational institutions to develop technical skills in combination with ethical behaviour and teamwork. Also, implementation of the CDX method has proved its worth as a proper competence evaluation and a motivating tool in cybersecurity-related military education (Schepens and James, 2003). It is a challenge to create cyber exercises equally stimulating for every participant. The passiveness of trainees who are either overwhelmed or insufficiently stimulated is a known problem (Henshel et al., 2016; Kick, 2014). Students may be involved in preparing the CDX game itself to overcome this problem (Svábenský et al., 2018). The involvement usually gives plenty of positive motivation to learn about new vulnerabilities and attack methods. Furtuna˘ et al. (2010) distinguish seven steps to create cybersecurity exercises for training purposes. The steps start with objectives and end with evaluation and lessons learned. Objectives. Tobey et al. (2014) point out that existing competitions are more attractive to experienced participants than to novices. Establishment of a preparatory environment to practice in advance of competitions might be a solution. Thus, the role and effectiveness of cybersecurity exercises should be discussed to find new cybersecurity talents and to encourage them to stay in the cybersecurity field. Wei et al. (2016) emphasise the need to educate general masses because they are the weakest link in cybersecurity defence during their everyday business activities. Therefore, CDX should address the needs of a broader audience than technical cybersecurity specialists. Vykopal et al. (2017) identified the general life cycle of a cyber defence exercise consisting of five phases with an emphasis on the preparation phase. The key success factor to the exercises is exercise difficulty matching the level of participants. Thus, during the preparation phase, the participants should be assessed or pretested to get their profile. This step could be implemented using surveys that include details about the employment, education, a self-assessment of cyber defence, security knowledge, and other exercise-related information (Henshel et al., 2016). An ideal person for cyber warfare operations is creative, having problem-solving skills, intelligent, and independent, but typically possessing such skills means a person does not tend to follow the rules well (Andress and Winterfeld, 2014). Team effectiveness is a critical element that determines success during cyber exercises (Buchler et al., 2018b). Steinke et al. (2015) present a multitude of practical recommendations for cyber response team performance improvement based on training experience of military, medical, and nuclear power plant operating teams. Adaptability, problem-solving, sharing team knowledge, trust building, and communication skills can be developed during pre-exercise training of participating teams. For example, adaptability could be improved by applying perturbation training. In addition to psychological challenges, each team also faces an environment overloaded with information. Team members use a variety of inter-team communication methods, information gathering and sharing tools, and

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

incident triage methods. These methods of team collaboration significantly impact the overall performance of the team (Rajivan and Cooke, 2017). Therefore, the methods should also be learned and trained. Buchler et al. (2018b) tried to find performance indicators by analysing collaboration and communication aspects. Greater faceto-face communication led to less effective team performance in tasks of service maintenance and incident response, although it was beneficial during scenario injects. Therefore, objectives of the exercises could be reached easier if functional role-specialisation was applied to construct teams (Buchler et al., 2018a) as the team results are influenced by proficiency, not by team size. Dawson and Thomson (2018) emphasise the importance of blending technical skills with social and cognitive skills to develop cybersecurity workforce. Thus, the scope of training and evaluation should be broadened to incorporate metrics of organisational and social fit. The objectives of the exercises could be defined using the existing security training and education standards (Dodge et al., 2009) supported by organisations such as ACM, ISO, NIST. Team types. Typically, cybersecurity exercises are oriented towards two teams: attackers and defenders (Furtuna˘ et al., 2010). Team types are distinguished based on an assigned colour. Blue teams work as cybersecurity response teams, and Red teams work as attackers. Other teams have their roles, too. In some international exercises, the White team represents exercise managers, referees, organisers, and instructors, and the Green team consists of operators and system administrators (Granåsen and Andersson, 2016; Vykopal et al., 2018). In Cyber Shield (related to US National Guard Bureau) class exercises (Henshel et al., 2016), EXCON team is distinguished to have a separate exercise control group, and White team consists of three members with different roles: Embedded Observer (EO), Training Analyst (TA), and Team Controller (TC) that are located in the space of Blue team, Red team, and EXCON, respectively. Sometimes additional team types are introduced, or they are assigned different roles. For example, a Grey team consists of White and Black in exercises conducted by students (Mauer et al., 2012) where the White team represents business operations, whereas the Black team supports the technical infrastructure. In National Collegiate Cyber Defence Competition (CCDC, 2019), Gold, White, and Black teams represent organisers, observers, and technical support, respectively. Consequently, there is no commonly accepted team naming standard apart from the Blue and Red teams. Assessment. The European Network and Information Security Agency reports that about half of cybersecurity exercises organised globally focus on training of participants and provide an opportunity to gain knowledge, understanding and skills. However, evaluation of individual or organisation capabilities and measuring of knowledge are not common (Ogee et al., 2015). Therefore, Seker and Ozbenli (2018) argue the importance of evaluation and scoring as a motivational and competition enabling instrument. Many researchers point out that the goal of most CDX is learning, but to assess individual participants or teams is a real challenge as objective tools and methodologies are lacking (Henshel et al., 2016; Maennel et al., 2017; Vykopal et al., 2018). In particular, the standard assessment based on voluntarily filled out questionnaires might be misleading and insufficient to identify the achieved proficiency during the exercises (Henshel et al., 2016). For some questions, self-assessment is more accurate than findings by observers, but it is still not a good predictor of performance during CDX (Granåsen and Andersson, 2016). Thus, surveys are treated as an unreliable performance measure. Vykopal et al. (2018) emphasise the need for a timely and individual group-oriented feedback during the exercises. The feed-

3

back stimulates learning and improves participant satisfaction during the exercises. Also, educators and organisers can use the feedback to tune the scenario to improve future exercises. Henshel et al. (2016) evaluated team dynamics using surveying method. The survey was completed by an embedded observer of the Blue team to address skills like collaboration, communication, leadership, and task distribution. The main parameters to assess technical skills were time to detect an incident and time to report it. Surveying could be used together with observation and analysis of data logs collected during the event (Granåsen and Andersson, 2016), although objective scoring was the main challenge. Maennel et al. (2017) created a 5-timestamp model enabling observers to assess group and individual skills within the Blue team, e.g. time management, task distribution, leadership. The assessment and evaluation are critical in cybersecurity education that is influenced by new technological advancements. Thus, Dark and Mirkovic (2015) present five-step recommendations to design evaluation in cybersecurity education to meet the needs of society. The evaluation must be framed by antecedents, transactions, and outcomes that are defined based on the underlying beliefs, assumptions, and theories. Cybersecurity society is interested in knowledge transfer, but education experts are rarely involved in cybersecurity training activities. Therefore, teaming of the two groups could enable the development of a reliable training system with an appropriate evaluation model to objectively track the progress of learners (Mirkovic et al., 2015). 3. Methodology We performed a case study of joint military-civilian cybersecurity exercises. The exercises aimed to improve international military-civilian cooperation in defending critical IT infrastructure. Training of the cybersecurity incident response team was the main focus as specified in MITRE guidelines (Kick, 2014). Planning and preparation of the scenario and the cyber-range lasted just under a year, and the execution took five full days. 3.1. Team setup and game rules The overall format of the exercise was of the Red-Blue (attack— defend) type. A separate identical network of over 30 virtual and physical machines was created for each participating team. The exercises were hybrid: defenders had to prevent attacks on their live infrastructure in real time, and they had to follow legal regulations while reporting the attacks to simulated authorities. Four colours were used to distinguish the participants. In Fig. 1 the colourwheel of different teams and their relationships to services and responsibilities is presented. White Team (WT) represents organisers, Red Team (RT) — attackers, Blue Team (BT) — defenders, and Purple Team (PT) simulates business owners. The White team was responsible for physical and virtual infrastructure management, exercise coordination (EXCON), and evaluation. Observers were a part of this team and had access to all information. Infrastructure as a service (IaaS) enabled the activities of every team. It was the responsibility of the White team to control the IaaS and provide technical support during the CDX. Several members of WT simulated the state Computer Emergency Response Team (CERT). They analysed team reports and acted according to the country’s legislation. The Red colour denoted skilled professionals who used automated and manual tools to perform attacks according to a predesigned exercise plan. One Red team was responsible for attacking every defending team. The Purple team represented employees of a simulated organisation and business end-users. They had to use networked services, IoT, ICS, and perform routine daily business operations, including

4

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

Fig. 1. An Overview of Teams and Their Responsibilities.

communication with external clients, and purchases and sales in a simulated marketplace. The team members also had all the necessary permissions to modify and change the services of their organisation as long as they would not hinder the business. At the same time, one Purple team acted as external business clients and performed actual daily business activities creating an additional burden for the internal PTs. This external PT was given a list of assignments they had to accomplish each day—orders to place, services to obtain. In other cyber exercises, these assignments would be called ”injects”, and they would be handed down to Blue teams to satisfy specific business needs in addition to defending the business. In our case, business operations and cyber defence operations were assigned to different teams. The business activities also generated data flows obscuring some of the attacks and making the scenario more realistic. Blue team acted as a cybersecurity incident response team (CSIRT) called to defend a simulated organisation. The formation of Blue teams was left for their appointed team leaders to decide. However, each of them fell under one of the following categories: • Ad hoc team members had no prior acquaintance and had little knowledge of personal or professional skills of each other. • Single organisation team members were from the same organisation and had prior knowledge of each other’s skills as well as previous working experience together. Teams of this type would usually have a leader with strong technical knowledge. • Hierarchical local team was a hierarchically organised and managed team with a formal leader, well-defined roles, understanding of each other’s responsibilities, personal and professional competencies from local homogeneous working places. The real-life working relationships among team members were cross-departmental with firm leadership. • Hierarchical international team was a hierarchically organised team with a good understanding of each other’s professional skills, but with an added international and multicultural aspect. Each BT had a separate room and working environment. The workplace had several TV screens and various devices to emulate real-life SCADA systems used in the critical infrastructure of the organisation with visible elements, e.g. railway traffic lights and alarm buzzers. The TVs were always showing output from different video cameras, SCADA systems, and the service and infrastructure monitoring tools. If any of the machines or services on the range became non-responsive, it was immediately apparent to the teams. In some cases, if specific critical services become unavail-

Fig. 2. CDX Timeline.

able, an annoying alarm would start beeping to increase the stress level of the participants. The monitoring software was also directly connected to the scoring system. The CDX started with detailed setup instruction and a short general introduction to participants. Each day the exercises started with a joint briefing where the teams had to recap the last day’s events and were presented an intelligence report about an everescalating situation in the region and country. In Fig. 2, the timeline of an exercise routine is shown. The exercise execution contained well recognisable stages balancing short breaks and actual work as well as information sessions. Daily briefings included intelligence information about future cyber-attacks and their types. WT organised the coordination meetings, and the defending teams were encouraged to have additional self-organised status checkpoints and share knowledge during the break and internal briefings. All blue teams had to decide on the set of elements of the infrastructure to focus their defensive attention on while maintaining prescribed business activities. Each simulated organisation had one PT and one BT working together and sharing the working environment. In Fig. 3, a general scheme of interaction of all participating teams is presented. A single external PT shown on the right of the figure served all simulated businesses. Other PTs cooperated with their corresponding BTs. The responsibility of BT and PT went beyond the deflection of the attacks. During an active defence phase, the joint team also had to analyse the attacks and correctly report them via specialised ticketing system to a simulated CERT within the WT. Well-constructed reports of one or other Blue/Purple team would sometimes induce the CERT to send warnings about the ongoing attacks to other Blue/Purple teams as in real-life situations.

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

5

Table 1 Pre-event questions for blue teams. No

Question for the Blue Team and Answer Type

1. 2. 3. 4.

Have you ever participated in similar exercises? Please, provide some details about the exercises, your role, your lessons learned. Open ended. How would you rate your skills, knowledge, and abilities in cybersecurity? Scale 1–5 (from Beginner to Professional) What are your expectations for the event? Check all that apply. Show off yourself. Have a good time. Establish contacts. Get a certificate. Other (please, specify). What are your strengths? Check all that apply. Firewall management. Usage of specialised software (to specify). Network analysis. Forensics. MS Windows OS. Linux OS. Database administration. Soft skills. Any other (please, specify). Any other comments. Open ended.

5.

Fig. 3. An Overview of CDX Team Interaction.

3.2. Observation and assessment Previous research of cyber exercises had pointed out a lack of objective evidence of their usefulness, especially regarding the learning experience of the participants. Therefore, we had placed observers in each team to monitor team collaboration and information sharing, identify the learning curves of the participants, note obstacles in maximising their learning experience as individuals or as a group. The observers had several tasks. Firstly, they needed to assess the participants and their expectations before the exercise. Secondly, they had to keep track of all individual activities from the perspective of teamwork during the five days noting all the observations in timed handwritten notes. Finally, they had to survey each participant at the end of the event. Additionally, they had an objective to try and map knowledge, skills, and abilities of every team member according to NICE roles based upon tools, commands and methods the observed participants used. Apart from the survey data and observation logs, observers also had in-depth interviews with representatives of White and Red teams. Blue team members were surveyed as the main focus was on the identification of the participant’s profile to know the training audience before the event. After the event, members of Blue, Red, and Purple teams were surveyed. We made a hypothesis that members of all teams achieved some learning progress. Thus, the survey was based on self-reflection on the usefulness of the event, challenges, used technologies. The participants were also asked to provide suggestions for improvement. Questionnaires were provided to participants to fill anonymously. Their relationships (team membership) among each other were not recorded. The pre-event questionnaire included five questions (see Table 1). The questions were related to prior experience (question 1), skill level (2 and 4), and expectations (3). The respondents could provide details about their previous roles in other cybersecurity exercises, specify personal strengths in the usage of technologies, and give some comments to organisers (5). A set of technical fields, e.g. firewall management, Linux OS, as well as the possibility to select soft skills, were listed in question 4 to narrow down the profile of Blue team members. There was a possibility to add and specify a specific professional skill. The list of technical fields was chosen after discussions with the WT to match the prepared attack plan.

The post-event questionnaires included a set of identical questions for all teams and a set of specific questions for different team types. The post-event questions are presented in Table 2. The first column of the table shows the question number and team(s) that received the question. Notation BPR means that the question was given to all teams. A dash was used if the question was not presented to the particular team. For example, notation B–R means that the Purple team did not receive the question. Questions 1–6 in Table 2 were designed to obtain general feedback regarding the exercises and the learning experience from all the participants. Blue and Purple teams had additionally to specify the technologies they used (7), indicate their knowledge gaps (8) and challenges they encountered during the exercises (9). Blue and Red teams reflected on their teamwork (10). They also could suggest technological solutions and cyber attack types for future exercises (11). Purple teams described the collaboration with Blue teams (12) and expressed their expectations about participation in future exercises (13). The Red team identified technologies, roles, and skills used during the exercises to develop their attacks (14). Note, that some Red team members had worked with the White team to prepare the infrastructure of the cyber exercises. Thus, some listed items are closely related to the preparation and support of the environment, e.g. hardware equipment, user support. Red team members could also reflect on their challenges during the preparation of attacks (15). On the third day of the exercises, the Observers performed an additional survey to measure the stress level of the Blue teams. This particular day was chosen because it represented normal working day conditions – during the first two days the teams had yet to get used to the environment and each other, and the last two days had planned interruptions because of visits by media and representatives of non-participating organizations. The third day also had a challenging scenario where the Red team had to hack into railway systems and derail a simulated train. Therefore both before, during, and after the main attack, each BT had to determine their stress level and fill the questionnaire 3 every hour. Later, the answers were supplemented with data from the observers and the timing of the attacks. 4. Results Data gathered during the exercise enabled us to measure the learning effect on the participants. The CDX event had more than 70 participants in total, including the organisers. There were four BTs (24 trainees), five PTs (14 trainees), and one RT (9 members). We obtained feedback from all of the participants, except 3 PT members. Results obtained from different teams cover several important areas related to exercise objectives. BT Profile. Typically, defence oriented CDX focus on training of BTs. The BT participant profile was constructed using collected data from the questionnaire presented in Table 1. Each participant selfevaluated proficiency in cybersecurity (five levels) and described the previous experience in CDX (number of times, roles, and CDX types). The constructed BT profile is presented in Fig. 4. The experience was classified into four colour-coded groups depending on

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

6 Table 2 Post-event questions. No.Type

Question and Answer Type

1. 2. 3. 4. 5. 6.

BPR BPR BPR BPR BPR BPR

Was it worth participating in the exercises? Yes or No How did the exercises meet your expectations? Scale 1–5 (from Poorly to Perfectly) What did you learn during the exercises? Open ended. What did you like in the exercises (atmosphere, particular incidents, etc.)? Why? Open ended. What did you dislike in the exercises? Why? Open ended. Any other comments, suggestions, feedback, ideas, thoughts that would help to organise and manage such an event (organisation, setup, etc.). Open ended.

7.

BP--

8.

BP--

9.

Which of the following have you used during the exercises? Check all that apply. Firewall management. Usage of specialised software (to specify). Network analysis. Forensics. MS Windows OS. Linux OS. Database administration. Soft skills. Any other (please, specify). Which of the following would you like to learn/improve? Check all that apply. Firewall management. Usage of specialised software (to specify). Network analysis. Forensics. MS Windows OS. Linux OS. Database administration. Soft skills. Any other (please, specify). Did you find any attack, task, or disruption challenging? If yes — which and why? Open ended.

10. 11.

How would you describe your teamwork? Did you have any problems? If yes — what? Open ended. What technological solutions or types of attacks might be considered next year? Open ended.

BP-B--R B--R 12. --P-13. --P-14. ----R

15.

----R

How would you describe your collaboration with a blue team? Did you have any problems? If yes — what? Open ended. Would you like to participate in the event next year? If yes—what team (red, blue, white, purple) would you like to choose? Open ended. Which of the following have you used during the exercises? Check all that apply. Network configuration, network maintenance, network monitoring and analysis, virtualisation, software installation, resource allocation, scripting, hardware/equipment installation, MS Windows OS, Linux OS, Database administration, firewall management, data analytics, forensics, programming, testing, server administration, scenario/algorithm design, soft skills, user support, usage of specialised software (specify), other (specify) Did you find the preparation of an attack, task, or disruption challenging? If yes — which and why? Open ended.

Table 3 Stress management self-evaluation form. Time: What is the stress level of the team How confident do you feel handling the situation? Explanation of the evaluation

Scale 1–5 (low to high) Scale 1–5 (not confident to very confident) Open ended

Fig. 5. Blue Team Participant Skills.

Fig. 4. Blue Team Participant Profile.

the number of previous exercises: no experience, 1-time experience, experienced (2–10 times), and very experienced (more than 10 times). Most of the participants self-assigned their proficiency to medium level and below (1–3). Similarly, over half of the participants had no previous experience in CDX. Only a few participants self-evaluated proficiency as high and very high (levels 4–5), and they were very experienced. Moderately experienced participants evaluated themselves at a medium proficiency (level 3). Among medium level participants, there were only a few with no experience, and the rest had participated in at least one CDX event. Thus, the audience was composed of novices in CDX with medium or lower proficiency in cybersecurity.

Skills of Participants. After the exercises, BTs and PTs answered questions concerning cybersecurity-related skills they used during exercises and skills they would like to learn afterwards (see Table 2). Additionally, BT members indicated skills they possessed before the exercises (see Table 1). The trends of skills are presented for BTs and PTs in Figs. 5 and 6, respectively. More than 50% of BT participants would like to improve their skills in firewall management, network analysis, forensics, and Linux OS. Only the development of skills in firewall management is relevant for more than 50% of PT members (approx. 70%). Approx. 35% of PT participants would be interested to learn network analysis, Linux OS, and database administration. Firewall, Linux OS, and forensics seem attractive to PT members even if these particular skills were not needed for Purple teams during the exercises. BT members were skilled in MS Windows OS and possessed soft skills. Also, more than 45% of BT and PT members indicated that soft skills and MS Windows OS skills were required, but they are not as attractive as other cybersecurity-related skills for further development. Fig. 7 presents the number of skills per individual for BTs and PTs for different time points of the exercises. Box plots show minimum and maximum numbers per individual, a median, and a standard deviation. BT participants had a small number of skills (with

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

Fig. 6. Purple Team Participant Skills.

Fig. 7. Number of Skills BT and PT Participants Claimed Before, Used During, and Expressed an Interest to Develop After the Exercises.

Fig. 8. PT Participants Expectations for the Next Year.

a couple of outliers), but they used more skills than they listed before the exercises. After the exercises, they created larger sets of skills to be developed. PT participants had to use a limited set of skills during the exercises. Afterwards, they expressed an interest to develop only a few, although completely new competencies. PT Ambitions. PT members answered the question if they would like to participate in CDX next year and what team they would choose. The answers to this question are presented in Fig. 8.

7

Out of all PT participants, 45% would like to be a part of BT only, and 18% would only select PT. Other respondents indicated that they would like to belong to any of the three teams. Out of 72% of participants that chose BT, 27% indicated BT or any other team. Out of 36% of participants selecting PT, 18% responded that the preference could be PT or some other team(s). No one answered that RT would be the only option for consideration. The results show that PT members are rather ambitious and consider themselves eligible for BT and even RT, but they do not single out a broad set of skills for further personal development as presented in Fig. 7. Some PT members experienced the exercises as not challenging enough. Also, several PT members think they could be more involved in BT work to learn more. Proper Start. Most of BT members emphasised the need for some pre-training before the exercises. The critical importance of preparation was also pointed out by other researchers (Maennel et al., 2017). The novices would expect some learning material and courses or lectures to have better orientation at the beginning of the exercises. A lack of preparations made a start confusing, e.g. missing information, unclear rules, as participants made assumptions based on their prior experience. The participants characterised the start as fast without proper analysis of the setting (network). Thus, participants treated exercises more as a training and learning event than a competition. Team Building and Collaboration. BT members highlighted that the exercises enabled the development of team collaboration and coordination (internally and externally) and the ability to work as a team. Task distribution was mentioned as a challenge and as an ability developed during the exercises. Participants mentioned a decision to divide themselves into subgroups to solve the tasks. Most BT respondents noted that their teams were perfect, team members were patient, helpful, and understanding. However, some BTs experienced miscommunication and problems with information sharing (insubordination) during the first days. A shortage of time allocated for team building was a negative aspect. Several respondents suggested prior team building exercises, especially for teams that contained members not knowing each other, e.g. ad-hoc and hierarchical international teams. The pre-training week could be used for team building as well as for ”touching the range”. Based on questionnaires, teaming was also important for RT and allowed expressing oneself. RT team was built on active core members who knew each other, but the members struggled with various difficulties and challenges, e.g. coordination, cooperation, a lack of human resources (e.g. for writing documentation), not enough time for attack testing, and keeping up with deadlines. However, they liked the stress, challenges, incidents, and the level of attack sophistication. Cross-Team Cooperation. PT members underlined they understood the importance of communication with team members and other organisations. Some PT participants would expect more accessible communication with the White team. BT members liked the exercise emphasis on communication, meeting new people during breaks, and access to other BTs/PTs. PTs highlighted an excellent atmosphere, challenges, and communication as positive aspects of the exercises. PTs mentioned some miscommunication with BTs at the beginning of exercises, but later their collaboration was excellent, friendly, and even fun. BT participants desired to obtain the technical details of the attack implementation. They liked briefings and missed more thorough and tech-styled explanations about the how and the what from the RT. Learning Curve. PT members emphasised they learnt a lot of new things about attacks and their impact on the system. They found it interesting to observe the effects of an infected file and to notice service perturbations. It was ”fascinating and yet scary how simple it is to break a network and how hard it is to restore it.” BT members marked that they improved technical skills, e.g.

8

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

Fig. 9. Red Team Participant Skills.

usage and configuration of monitoring tools, network analysis, and incident analysis. The attacks were described as challenging, not obvious, exciting, fun, and difficult at the same time. Stress management self-evaluation form (see Table 3) revealed that all teams except the ad hoc team experienced low-stress level throughout the exercises. The confidence level for most teams increased during the period of intensive attacks, even when they were struggling with unavailable services. The ad hoc team had decreased confidence during the attack, and one team had a high confidence level (4 out of 5) throughout the stress observation. Half of the teams failed to protect their systems, but their reported stress level remained very low. Thus, their self-confidence did not correspond to the performance. Granåsen and Andersson (2016) had reached a similar conclusion about self-assessed expertise being a poor predictor of performance. The BT participants liked the variety of attacks. They also suggested several improvements for future exercises: more noise in the range, higher attack speed (frequency), a larger number of attacks, tasks in forensic and social engineering, a spy in a blue team that would try to sabotage the team, and cooperation with other BTs as a mandatory element to solve some tasks. PTs enjoyed the element of unexpectedness, a possibility to learn web system administration, and similarly to BTs, they would like more social engineering aspects. RT members mentioned they gained experience in organising the event, planning the overall picture, and understanding psychology of the blue team. They made new connections and improved critical thinking. Fig. 9 presents skills that were needed by RT members. The majority of RT members performed tasks requiring network maintenance, software installation, virtualisation, scripting, Linux, and server administration. They had challenges with the scenario and attack coordination, virtualisation and configuration. In questionnaires, RT members mentioned they learnt new management tools and new attack methods. Also, they identified areas for future personal development. They used soft skills and had to provide user support. Therefore, participation in RT provides opportunities for self-assessment and gaining experience in the application of technical and soft skills. According to the majority of respondents, regardless of the team type, the general atmosphere and environment of the exercises enabled learning, self-reflection, and improvement in technical and soft skills. The results are consistent with Observer reports and insights from an interview with RT members. The exercise

increased the awareness of individuals regarding their knowledge and skill gaps. 5. Discussion We did a case study on a hybrid CDX to find out if this type of event is optimal for cybersecurity competence development and assessment. CDX observers performed surveys, interviews, and continuous observation of participants focusing on learning activities, collaboration, and self-assessment. The methodology encouraged self-reflection and helped to answer our research question. We present several CDX improvements based on our findings. Proper arrangement of CDX could enable the development of competencies of all involved participants. CDX is a resourceintensive event, and it could be used more effectively than just for a specific group of individuals. Our results show that Purple Team members also gain knowledge about cybersecurity, and they would like to be involved more actively than usual. PT represents business users, decision makers, managers, and fresh or non-technical users whose actions may have a considerable impact on cyber defence in any organisation. Thus, organisers could include specific tasks for PT in the scenario to increase their cybersecurity awareness. Objectives of CDX should cover all participating groups. Specific learning outcomes should be defined for the defenders (Blue Team), attackers (Red Team), business users (Purple team), and even infrastructure support specialists (White Team). The outcomes should not necessarily be related to technical skills. During our case study, Red Team members pointed out their soft skill development, e.g., the psychology of Blue Teams, collaboration, and time planning, even though they were not the main target group of the exercises. List of competencies to be trained should be derived directly from CDX objectives and not from the assigned roles of team members. During the analysed CDX, a specific shortened list of the relevant NICE framework roles had been made before the exercise. The Observers were given a task to match the roles to each of the participants, but despite a considerable effort, they failed. Many participants exhibited skills and knowledge assigned to many different roles with a blurred overlap. The reason was a broad spectrum of attacks and a variety of used technologies. An overabundance of possible safeguarded targets forced members of limited-size teams to assume different roles during different attacks. Therefore, to

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

9

Fig. 10. Detailed Layout of Training Phases.

train all competencies of particular roles, the scenario of attacks should be adapted correspondingly. For example, the organisers should choose only particular attack types and then observe and evaluate specific steps performed by a participant playing a particular role. Alternatively, a specific subset of competences of chosen roles can be selected and developed according to the objectives of the CDX. CDX time-flow should include dedicated points to assess learning outcomes derived from the objectives for different types of teams. According to scientific literature, attempts to measure the performance of participants had considerable challenges and uncertainties. During CDX, team performance can be based on many criteria. Measured data points represent a single moment result, but not the impact of the whole CDX on competence development. Tracking of an individual learning curve is even more challenging as it requires substantial resources. Also, a team performs successfully if at least one member has the necessary skills to solve an incident or prevent an attack. Personal learning value of the CDX mostly comes from anecdotal evidence. Thus, time should be allocated for assessment separately from the training activities. Pre-training should become an integral part of the exercise to fill the competency gap for all individual participants. First of all, each participant needs skills outside the specialisation or previous experience. Secondly, CDX is a team based event. Thus, even professionals should participate in team building activities before the main competition. Pre-training would also serve as a platform for initial assessment of the audience. It would aid as an intrinsic motivation tool and as a baseline for learning curve measurements. To sum up our findings, we propose an extended competence development and assessment (CDA) framework. The CDA framework combines methods, tools, and procedures to develop and assess competences of all CDX participants including non-technical players. The framework supplements a typical CDX life cycle to enable competence assessment at an individual level to reach learning objectives. It consists of four phases detailed in Fig. 10. Phase 1 is dedicated to Pre-exercise assessment of the training audience and design of learning objectives. The learning objectives of the CDX depend on the audience type (proficiency, experience, specialisation), and competence map might be different for each team type (Red, Blue, Purple, White). During Phase 1, the learning objectives are adjusted based on the CDX goals and the identified participant audience profile. Phase 2 covers pre-exercise training of teams and individuals. An individual participant may be trained based on personal competence matrix and task specifics for the role assigned in a team. The basis for efficient teamwork is built by organising various group activities. For example, team building would be beneficial to ad-hoc teams. If CDX objectives include evaluation of

Fig. 11. General Incident Flow During CDX.

performance under stress, then experience gathered from medical or other emergency response teams can be applied by designing special pre-training exercises. Also, pre-training should specifically cover the legal base to emphasise the importance of comprehensive reports. Instructions on how to use the reporting system should be provided, as participants tend to give low priority to the reporting task. Pre-exercise training should include an assessment strategy to follow the learning progress of participants. Phase 3 is a standard main component of CDX where pretrained teams play against each other as well as train to respond to incidents by facing unpredictable non-routine challenges. Typical exercises are competitive, and training activities are underused. Phase 3 should include hot wash-up covering more details than usual to cover technical details about attack implementation and defence strategies. As a rule, each team operates in an isolated environment. During hot wash-up, one team can be chosen to present their defence strategy against a specific attack. Thus, teams can share their best practices and learn from each other. Some attacks can be periodically repeated several times to objectively assess performance improvement and help the teams achieve learning outcomes. Simulation of a real-world environment serves as a motivational tool. Some time of CDX should be allocated for ”competitive exercises” even when the primary goal of CDX is training. A scoring system encourages competition atmosphere. We suggest a scoring

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

10

Table 5 An example mapping of exercise objectives to exercise phases.

Table 4 A description of timestamps from Fig. 11.

[T1] [T2] [T3] [T4] [T5] [T6] [T7] [T8]

Action

Objective

P1

P2

P3

P4

WT orders RT to initiate a cyber attack. RT launches the attack and records the result. PT notifies BT of service failure, or BT detects the attack. BT evaluates the threat level and sends a short report. BT defends and recovers the system. BT submits detailed report of the incident with indicators of compromise and possible attribution if possible. WT orders RT to validate the previous attack. RT reports the final status of the attack.

Apply tool X for network monitoring Ensure web-server availability Recover service after crash  Perform incident triage

  

  

  

  









system covering both attack-defend actions and reports by extending the 5-timestamp model (Maennel et al., 2017) by adding three additional points (T6–T8). The schema of the typical workflow of an incident from activation to resolution is depicted in Fig. 11. WT initiates incidents according to the CDX scenario. RT launches the attack, and then PT together with BT should detect it, take defensive actions, investigate the incident, and report back to WT (see Table 4). Timestamps of each step of the incident flow can be recorded either automatically or by Observers. Interval lengths can be scored to measure team (and sometimes—individual) performance as described by Maennel et al. (2017), e.g., T3–T2 measures Time to Detect, and T5–T2 corresponds to Time to Restore. Validation of the recovered service (steps T7, T8) could even be performed during another incident independently of the current status of the system as well as multiple times. During hybrid CDX, the game does not end with an incident resolution and requires additional efforts from the BT to perform incident triage and describe the process in detail (T6). WT can additionally score the quality of the reports as well as use them during hot wash-up sessions to educate less successful teams. Phase 4 of CDX in our framework is dedicated to the postexercise assessment of competencies of individual participants because Phase 3 can mostly be used to determine team performance. If CDX is implemented as a part of formal education, then the most common assessment strategy with assessment criteria can be applied to grade the students using tests and practical assignments as assessment methods. Alternatively, if CDX is organised to train professionals, then less formal assessment methods such as questionnaires would support self-reflection about the development of skills. Assessment results from all phases are compiled into the final assessment report giving a comprehensive view on the impact of the event both to individual participants and to each team. Learning objectives are defined in Phase 1 of CDX with many assessment points during the four phases of the exercise. Assessment methods should match the assessment strategy and criteria based on learning objectives. We suggest using a competence planner to keep track of the objectives during all the phases as presented in Table 5. Each objective should be mapped to at least one

phase where the participants are trained and assessed to achieve a corresponding learning outcome. The number of sub-phases could be easily adapted to support individual and team competence learning objectives. The CDA framework complements the traditional CDX life cycle having four stages: Identify, Plan, Conduct, and Evaluate. In Fig. 12, the mapping of the CDX life cycle to the CDA phases is shown, and key results are presented in the corresponding stage and phase. As depicted in the figure, CDA phases are spanning over the whole CDX life cycle. Usually, training and assessment are concentrated in the Conduct stage. However, CDA activities should start in the Identify stage and continue past LiveEx to benefit most from the CDX. The presented layout of CDA activities enables a learneroriented approach. Learning objectives and assessment strategies are adapted to the training audience based on the participant profile. Also, the learning objectives can be differentiated for technical and non-technical participants. Results of the post-exercise assessment phase can be integrated into CDX first impression and final reports as a measurable indicator of the CDX value. A detailed timeline of CDA framework activities is described in Table 6. The first and last columns indicate periods of CDX life cycle stages and CDA phases, respectively (corresponding to Fig. 12). The other two columns describe and explain suggested steps, actions, responsibilities, and tools with a clear assignment to the phases. The table specifies only steps related to competence development and assessment, while other standard CDX life cycle activities are omitted (see MITRE Playbook for their list (Kick, 2014)). The CDA activities require extra resources, and the White Team takes on additional responsibilities. Therefore, an Educational Team (EduTeam) is created as a part of the White Team. The EduTeam mainly consists of observers and academic personnel, and it closely collaborates with the EXCON. The EduTeam selects a suitable competence framework, prepares a pre-training plan for all participants, and creates a competence development and assessment schema corresponding to the defined CDX concept and objectives. Competence frameworks are extensive. Thus, the EduTeam should identify key competences to be developed during the CDX. For example, according to the NICE framework (Newhouse et al., 2017) a person in the database administrator role should be able to perform more than ten tasks and possess knowledge and skills in over twenty topics and areas. Some of them cannot be easily developed

Fig. 12. A High-level Overview of the CDA Framework.

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

11

Table 6 A detailed timeline of the CDA framework.

IDENTIFY

Step

Description, examples, notes

Concept development meeting. Key stakeholders define CDX concept, objectives, and participant profile.

The White team determines size and type of CDX, possible training audience, and other essential parameters.

EduTeam selects reference competence frameworks for technical and non-technical participants.

The NICE framework might be used for IT persons, ACM/IEEE—for students in a higher education institution, and information or cybersecurity curricula—for non-IT persons. A subset of competencies to be developed during the CDX is chosen from the selected frameworks. Then, the subset is narrowed for detailed assessment. Different parts of CM may apply to different teams and individual roles.

EduTeam identifies roles, tasks, and competencies to be trained and assessed based on the CDX objectives and creates a competence map (CM). PLAN

CONDUCT

EVALUATE

Phase 1

Initial planning meeting. White Team supplies an initial participant list. Stakeholders provide feedback on CM.

Note, the list of participants might be incomplete at this stage.

EduTeam prepares and uses remote tests and/or questionnaires to profile participants.

Assessment tools and methods (e.g., virtual learning environment, test cyber ranges, or online questionnaires) are used to determine the current level of participants according to CM.

EduTeam updates CM and defines learning objectives (LOs).

Based on the obtained results, EduTeam defines LOs using Bloom, SOLO, or other taxonomy.

Main planning conference. EduTeam and White Team coordinate requirements for the environment, range, attack vectors, and scenario to enable development and assessment of competences according to the set LOs.

CM should be fully covered at this point. Specifically, the scenario should address the training of the PT. Tools are selected to facilitate automated and semi-automated scoring of teams. Appropriate legislative documents, rules of engagement, and technological solutions applied during the CDX are determined during this step.

EduTeam makes a plan for pre-training, selects assessment methods and criteria for Pre-exercise training, LiveEx, and Post-exercise assessment phases.

Pre-training schema is selected. EduTeam prepares tools for the assessment, e.g. questions and tasks for the pre-exercise activities of all teams (and roles, if applicable) with assessment criteria in the virtual learning environment.

Final planning conference. EduTeam finalises assessment methods. White Team provides participant lists.

EduTeam and White Team define what group and individual tasks of PT, BT, and RT will be used in the assessment process. Specifically, RT might get additional tasks during LiveEx.

EduTeam initiates pre-training (moderate intensity, self-driven, off-site).

The learning/training material (e.g. documentation, instructions, information on remote access to test range, first tasks) is provided to participants based on the pre-training schema.

EduTeam continues intense pre-training on-site. Participant/team learning progress is assessed using selected tools. Team profiles are created.

Pre-training schema is implemented. Different teams might get specific training that corresponds to the CDX scenario. E.g. PTs study cases on how to avoid cybersecurity incidents by implementing security policies and how to report detected incidents in the context of business processes and social issues of the organisation.

LiveEx. CDX progresses according to Fig. 2 timeline. White Team monitors, scores, observes and assesses teams, gathers factual observation data.

The assessment and scoring schema is implemented. Observers make notes and log participant activities/behaviour based on the assessment criteria, e.g. PT ability to follow the organisation security policy described in the playbook, and BT ability to re-define roles or take leadership after removal of a member. Repeated attacks could be implemented to observe learning progress.

Phase 3

White Team organises the collection of participant self-reflection on the CDX.

A self-reflection session should be organised right before the EndEx to collect fresh impressions.

Phase 4

EduTeam prepares the final assessment report and issues competence certificates. White Team integrates the assessment results into the first impression and final reports.

Analysis of the log and observation data is performed to make the final assessment and grading. Additional scoring points might be given for tasks completed by teams and/or competences shown. In formal education, an exam might be organised.

and assessed during the CDX (e.g. Provide recommendations on new database technologies and architectures), others are off-topic regarding the CDX objectives (e.g. a skill in optimising database performance). Usually, LiveEx requires a lot of different competences to deal with challenges during the attacks. However, it would require huge resources to develop and assess each of them individually. Hence, the EduTeam selects key competences during the Identify stage and narrows down the list during the Planning stage. The short-listed competences result in learning objectives and competence development and assessment schema (tools and methods): pre-training tasks and type, material, lectures, case studies, test questions, questionnaires, assessment criteria for level identification, attack sequence and implementation (e.g. re-launch), learning/training environment (e.g. virtual learning environment, cy-

Phase 2

ber range implementation), and post-exercise assessment type (e.g. exam). The EduTeam collaborates with all other White Team subgroups and influences preparation of the scenario to cover the competence development needs of all team types, especially the non-technical participants. Therefore, CDA framework enables to develop, objectively assess, and certify competences of all CDX participants. 6. Conclusions Acquisition of cybersecurity skills and abilities requires commitment and time. Exercises aimed at developing cyber skills are also time-consuming and costly events that require dedication and collaboration of trainers and trainees. Our analysis of related works

12

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite,

and our observations showed that purely competitive exercises do not satisfy the learning needs of every participant and fail to measure their learning progress. To optimise the training output, we presented a framework addressing the competence development needs of a broader exercise audience. The framework partially sacrifices the competitive nature of CDX. We recommend all teams to spend more time analysing, reporting and reflecting on attacks. Several assessments and focused training stages embedded within the exercise would enable the measurement of the learning curve and add additional motivation for participants. The framework makes it possible to use CDX effectively both in professional training and formal education. Also, the initial evaluation of participants before the exercise allows the organisers to know the training audience better and suggest possible ways to fill in knowledge gaps. With such a framework, novices would not be overwhelmed with complexity and would be motivated to stay in the field of the cybersecurity. 7. Future work Future work can have several research directions related to competence development of cybersecurity specialists and organisation of CDX. Firstly, future studies could explore the application of the CDA framework in different settings and a broad audience of participants. For example, the timeline points (steps of EduTeam), dominant competences, pre-training duration, and proportion of the individual and group training in formal education and professional training might be determined, compared, and optimised. Further investigation in the design of teams is necessary to identify an optimal team composition to perform assigned tasks effectively. We envision early mappings of individual competences would aid in the team composition, but the criteria should be analysed and identified based on the organisational environment, attack types, and other relevant parameters. Also, an exciting research topic would be to determine the influence of the number of high-level professionals on the learning curve and motivation of other team members (e.g. ad-hoc novices). Alternatively, future research could consider the effect of inclusion of non-technical people in BTs, e.g. a scenario could provide these participants with extra facts to simulate an increased situational awareness. The CDA framework would benefit from a set of semiautomated tools designed to generate a skeleton of the scenario, attack vectors, tasks, and assessment criteria based on the competence map chosen by the CDX organisers. A generator algorithm could be created after specific case studies investigating combinations of roles, attacks, participant competences, and rate of involvement of non-technical trainees. Finally, future research could cover the analysis and development of tools to facilitate the assessment of the trainee performance during exercises and objectively evaluate the learning curve. Our exercise competence planner can be tuned according to the team composition types. It could be extended with assessment criteria for each trained competence to follow the learning progress of an individual or a team. Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Acknowledgements The authors of the paper would like to express their gratitude to the organisers and participants of the international cy-

bersecurity exercise ”Amber Mist2018” for the opportunity to observe and gather data. The authors are also thankful for the computing resources provided by the IT Research Center of Vilnius University. References Andress, J., Winterfeld, S., 2014. Cyber Warfare—Techniques, Tactics and Tools for Security Practitioners, 2nd Syngress, an imprint of Elsevier doi:10.1016/ C2013- 0- 0 0 059-X. Aoyama, T., Nakano, T., Koshijima, I., Hashimoto, Y., Watanabe, K., 2017. On the complexity of cybersecurity exercises proportional to preparedness. J. Disaster Res. 12 (5), 1081–1090. doi:10.20965/jdr.2017.p1081. Buchler, N., La Fleur, C.G., Hoffman, B., Rajivan, P., Marusich, L., Lightner, L., 2018a. Cyber teaming and role specialization in a cyber security defense competition. Front. Psychol. 9 (2133), 17. doi:10.3389/fpsyg.2018.02133. https://www. frontiersin.org/article/10.3389/fpsyg.2018.02133 Buchler, N., Rajivan, P., Marusich, L.R., Lightner, L., Gonzalez, C., 2018b. Sociometrics and observational assessment of teaming and leadership in a cyber security defense competition. Comput. Secur. 73, 114–136. doi:10.1016/j.cose.2017.10.013. http://www.sciencedirect.com/science/article/pii/S0167404817302298 Dark, M., Mirkovic, J., 2015. Evaluation theory and practice applied to cybersecurity education. IEEE Secur. Privacy 13 (2), 75–80. doi:10.1109/MSP.2015.27. Dawson, J., Thomson, R., 2018. The future cybersecurity workforce: going beyond technical skills for successful cyber performance. Front Psychol 9, 744. doi:10. 3389/fpsyg.2018.00744. Dodge, R.C., Hay, B., Nance, K.L., 2009. Standards-based cyber exercises. In: Proceedings of the The Forth International Conference on Availability, Reliability and Security, ARES 2009, March 16–19, 2009, Fukuoka, Japan. IEEE Computer Society, pp. 738–743. doi:10.1109/ARES.2009.72. European Union Agency for Network and Information Security (ENISA), 2016. NCSS Good Practice Guide Designing and Implementing National Cyber Security Strategies. Technical Report. Publications Office of the European Union doi:10. 2824/48036. TP-05-16-002-EN-N, ISBN: 978-92-9204-179-3 European Union Agency for Network and Information Security (ENISA). Cyber Europe 2018—After Action Report Findings from a cyber crisis exercise in Europe; 2018. TP-06-18-410-EN-N, ISBN: 978-92-9204-287-5; 10.2824/369640. European Union Agency for Network and Information Security (ENISA). ENISA Threat Landscape Report 2018; 2019.. 10.2824/622757. Furtuna˘ , A., Patriciu, V.V., Bica, I., 2010. A structured approach for implementing cyber security exercises. In: 8th International Conference on Communications. IEEE, pp. 415–418. doi:10.1109/ICCOMM.2010.5509123. Granåsen, M., Andersson, D., 2016. Measuring team effectiveness in cyber-defense exercises: a cross-disciplinary case study. Cognition, Technology & Work 18 (1), 121–143. doi:10.1007/s10111- 015- 0350- 2. Henshel, D.S., Deckard, G.M., Lufkin, B., Buchler, N., Hoffman, B., Rajivan, P., Collman, S., 2016. Predicting proficiency in cyber defense team exercises. In: MILCOM 2016 - 2016 IEEE Military Communications Conference, pp. 776–781. doi:10.1109/MILCOM.2016.7795423. Hoffman, L.J., Rosenberg, T., Dodge, R., Ragsdale, D., 2005. Exploring a national cybersecurity exercise for universities. IEEE Security & Privacy 3 (5), 27–33. doi:10.1109/MSP.2005.120. Joint Task Force on Cybersecurity Education, 2017. Cybersecurity Curricula 2017: Curriculum Guidelines for Post-Secondary Degree Programs in Cybersecurity. Technical Report. ACM/IEEE/AIS-SIGSEC/IFIP WG 11.8, New York, NY, USA doi:10. 1145/3184594. Kick, J., 2014. Cyber exercise playbook. Technical Report. MITRE Corp Bedford, MA, USA. https://apps.dtic.mil/dtic/tr/fulltext/u2/a624910.pdf Maennel, K., Ottis, R., Maennel, O., 2017. Improving and measuring learning effectiveness at cyber defense exercises. In: Lipmaa, H., Mitrokotsa, A., Matulevicˇ ius, R. (Eds.), Secure IT Systems - 22nd Nordic Conference, NordSec 2017, Tartu, Estonia, November 8–10, 2017, Proceedings, volume 10674. Springer, pp. 123–138. doi:10.1007/978- 3- 319- 70290- 2_8. Lecture Notes in Computer Science Mauer, B., Stackpole, B., Johnson, D., 2012. Developing small team-based cyber security exercises. In: Proceeding of the 2012 International Conference on Security and Management (SAM’12), pp. 213–217. Las Vegas, NV, USA Mirkovic, J., Dark, M., Du, W., Vigna, G., Denning, T., 2015. Evaluating cybersecurity education interventions: three case studies. IEEE Secur. Priv. 13 (3), 63–69. doi:10.1109/MSP.2015.57. Morgan S. Top 5 cybersecurity facts, figures, predictions, and statistics for 2019 to 2021. Cybersecurity Ventures, Cybercrime magazine; 2019. National CCDC. Collegiate cyber defense competition. 2019. http://www. nationalccdc.org/index.php/competition/competitors/rules. Newhouse, W., Keith, S., Scribner, B., Witte, G., 2017. National initiative for cybersecurity education (NICE) cybersecurity workforce framework. NIST Spec. Publ. 800-181, 144. doi:10.6028/NIST.SP.800-181. Ogee, A., Gavrila, R., Trimintzios, P., Stavropoulos, V., Zacharis, A., 2015. The 2015 Report on National and International Cyber Security Exercises. Technical Report. European Network and Information Security Agency doi:10.2824/627469. ISBN: 978-92-9204-158-8 Ohta, T., Takenaka, M., Katou, M., Masuoka, R., Kayama, K., Fukushima, N., Imai, H., 2018. Cybersecurity solutions for major international events. Fujitsu Sci. Tech. J. 54 (4), 57–65.

˙ L. Bukauskas and A. Juozapavicˇ ius / Computers & Security 88 (2020) 101607 A. Brilingaite, Parrish, A., Impagliazzo, J., Raj, R.K., Santos, H.M.D., Asghar, M.R., Jøsang, A., Pereira, T., Stavrou, E., 2018. Global perspectives on cybersecurity education for 2030: a case for a meta-discipline. In: Rößling, G., Scharlau, B. (Eds.), Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE 2018, Larnaca, Cyprus, July 02–04, 2018. ACM, pp. 36–54. doi:10.1145/3293881.3295778. Paulsen, C., McDuffie, E., Newhouse, W., Toth, P., 2012. NICE: Creating a cybersecurity workforce and aware public. IEEE Security & Privacy 10 (3), 76–79. doi:10.1109/MSP.2012.73. Rajivan, P., Cooke, N.J., 2017. Impact of team collaboration on cybersecurity situational awareness. In: Liu, P., Jajodia, S., Wang, C. (Eds.), Theory and Models for Cyber Situation Awareness, volume 10030. Springer, pp. 203–226. doi:10.1007/ 978- 3- 319- 61152- 5_8. Lecture Notes in Computer Science Schepens, W., James, J.R., 2003. Architecture of a cyber defense competition. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance, volume 5, pp. 4300–4305. doi:10.1109/ICSMC.2003.1245660. Seker, E., Ozbenli, H.H., 2018. The concept of cyber defence exercises (CDX): planning, execution, evaluation. In: 2018 International Conference on Cyber Security and Protection of Digital Services, Cyber Security 2018, Glasgow, Scotland, United Kingdom, June 11–12, 2018. IEEE, pp. 1–9. doi:10.1109/CyberSecPODS. 2018.8560673. Steinke, J., Bolunmez, B., Fletcher, L., Wang, V., Tomassetti, A.J., Repchick, K.M., Zaccaro, S.J., Dalal, R.S., Tetrick, L.E., 2015. Improving cybersecurity incident response team effectiveness using teams-based research. IEEE Secur. Privacy 13 (4), 20–29. doi:10.1109/MSP.2015.71. Svábenský, V., Vykopal, J., Cermák, M., Lastovicka, M., 2018. Enhancing cybersecurity skills by creating serious games. In: Polycarpou, I., Read, J.C., Andreou, P., Armoni, M. (Eds.), Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE 2018, Larnaca, Cyprus, July 02–04, 2018. ACM, pp. 194–199. doi:10.1145/3197091.3197123. Tobey, D.H., Pusey, P., Burley, D.L., 2014. Engaging learners in cybersecurity careers: lessons from the launch of the national cyber league. ACM Inroads 5 (1), 53–56. doi:10.1145/2568195.2568213. Vykopal, J., Oslejsek, R., Burská, K., Zákopcanová, K., 2018. Timely feedback in unstructured cybersecurity exercises. In: Barnes, T., Garcia, D.D., Hawthorne, E.K., Pérez-Quiñones, M.A. (Eds.), Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE 2018, Baltimore, MD, USA, February 21– 24, 2018. ACM, pp. 173–178. doi:10.1145/3159450.3159561.

13

Vykopal, J., Vizváry, M., Oslejsek, R., Celeda, P., Tovarnák, D., 2017. Lessons learned from complex hands-on defence exercises in a cyber range. In: 2017 IEEE Frontiers in Education Conference, FIE 2017, Indianapolis, IN, USA, October 18–21, 2017. IEEE Computer Society, pp. 1–8. doi:10.1109/FIE.2017.8190713. Wei, W., Mann, A., Sha, K., Yang, T.A., 2016. Design and implementation of a multifacet hierarchical cybersecurity education framework. In: Proceedings of IEEE Conference on Intelligence and Security Informatics (ISI). IEEE, pp. 273–278. doi:10.1109/ISI.2016.7745488. White, G.B., Dietrich, G.B., Goles, T., 2004. Cyber security exercises: testing an organizations ability to prevent, detect, and respond to cyber security events. In: Proceedings of the 37th Hawaii International Conference on System Sciences (HICSS-37 2004), CD-ROM / Abstracts Proceedings, 5–8 January 2004, Big Island, HI, USA. IEEE Computer Society, pp. 1–10. doi:10.1109/HICSS.2004.1265411. Agne˙ Brilingaite˙ . Brilingaite˙ holds a PhD in computer science from Aalborg University, Denmark. She is an associate professor at Vilnius University in the Institute of Computer Science. Her research interests focus on spatial data modelling, locationbased services, cybersecurity training, and education in computer science. She is involved in the process of quality assurance in studies at the university. She has been taking part in EU-funded projects related to the development of student-centred learning, teaching, assessment, and internationalisation. Linas Bukauskas. Bukauskas holds a PhD in computer science from Aalborg University, Denmark. He is an associate professor and head of Cybersecurity Laboratory in the Institute of Computer Science at Vilnius University. He was one of the organisers of National Cybersecurity Training “Cyber Shield” and “Amber Mist” (2016– 2018). His research interests include Cybersecurity, Data Mining, and Natural Language Processing. Aušrius Juozapavicˇ ius. Juozapavicˇ ius holds a PhD in theoretical physics from KTH Royal Institute of Technology, Sweden. He is a professor and the head of the Department of Defence Technologies at General Jonas Žemaitis Military Academy of Lithuania. His research interests are cybersecurity and computer modelling and optimisation of various systems, including semiconductor antennas and road traffic. He participates in EU-funded cybersecurity-related projects, and he is responsible for the cybersecurity specialisation of the study programs at the Military Academy.