Validating the fundamental cybersecurity competency index (FCCI) through expert evaluation for human-generative artificial intelligence (GenAI) teaming
Abstract
The increasing volume of cyber threats, combined with a critical shortage of skilled professionals and rising burnout among practitioners, highlights the urgent need for innovative solutions in cybersecurity operations. Generative Artificial Intelligence (GenAI) offers promising potential to augment human analysts in cybersecurity, but its integration requires rigorous validation of the fundamental competencies that enable effective collaboration of human-GenAI teams. Fundamental cybersecurity competencies, encompassing essential cybersecurity Knowledge, Skills, and Tasks completion (KSTs). Competency is defined as the ability to complete tasks within a work role. In this research study, we employed a mixed-methods research approach designed to evaluate human-GenAI teams, emphasizing the role of expert consensus in shaping the experimental assessment of the Fundamental Cybersecurity Competency Index (FCCI) in a commercial cyber range. Selecting a commercial cyber range allowed us to identify the specific KSTs from the United States (U.S.) Department of Defense (DoD) Cyber Workforce Framework (DCWF) and measure them at the KSTs level. The specific commercial cyber range we assessed enables the extraction of the users’ performance at the KST level. To validate the proposed experimental assessment of the FCCI and confirm the relevance of the selected cybersecurity KSTs, a panel of 20 Subject Matter Experts (SMEs) was engaged to evaluate and validate the proposed competency measures. The expert panel refined the cybersecurity scenarios and experimental procedures used in the commercial cyber range hands-on scenarios, ensuring alignment with DCWF. Our findings indicated that 46 of 47 fundamental cybersecurity KSTs were validated by the SMEs as essential components of the FCCI. Consensus levels of 85–90% confirmed strong expert support for incorporating GenAI (e.g., large language models such as ChatGPT) as a teammate or decision-support agent in these controlled experiments. The validated scenarios and experiments pave the way for future research on assessing cybersecurity competencies in commercial cyber range platforms with and without GenAI support (e.g., large language models such as ChatGPT). By establishing the baseline for competency assessment in this research, the SMEs’ feedback contributed to advancing cybersecurity workforce development and provided critical insights for integrating GenAI into collaborative cybersecurity human-GenAI teaming operations. The validated FCCI provides a robust mechanism to evaluate both human and human–GenAI team performance within realistic cybersecurity scenarios, while providing the needed metrics to measure cybersecurity competencies quantitatively. While this study achieved strong consensus, like any other research, several limitations were observed, including a relatively small SME panel size (n=20) and the absence of empirical testing with users. Future research will employ hands-on cyber range experiments to measure the FCCI by comparing KSTs measured across human-only and human–GenAI teams. Ultimately, this research advances cybersecurity workforce development by establishing a validated foundation for a quantitative assessment of cybersecurity competencies based on DCWF necessary for effective collaboration between humans and GenAI in defending against complex and evolving cyber threats.
Validating the fundamental cybersecurity competency index (FCCI) through expert evaluation for human-generative artificial intelligence (GenAI) teaming
The increasing volume of cyber threats, combined with a critical shortage of skilled professionals and rising burnout among practitioners, highlights the urgent need for innovative solutions in cybersecurity operations. Generative Artificial Intelligence (GenAI) offers promising potential to augment human analysts in cybersecurity, but its integration requires rigorous validation of the fundamental competencies that enable effective collaboration of human-GenAI teams. Fundamental cybersecurity competencies, encompassing essential cybersecurity Knowledge, Skills, and Tasks completion (KSTs). Competency is defined as the ability to complete tasks within a work role. In this research study, we employed a mixed-methods research approach designed to evaluate human-GenAI teams, emphasizing the role of expert consensus in shaping the experimental assessment of the Fundamental Cybersecurity Competency Index (FCCI) in a commercial cyber range. Selecting a commercial cyber range allowed us to identify the specific KSTs from the United States (U.S.) Department of Defense (DoD) Cyber Workforce Framework (DCWF) and measure them at the KSTs level. The specific commercial cyber range we assessed enables the extraction of the users’ performance at the KST level. To validate the proposed experimental assessment of the FCCI and confirm the relevance of the selected cybersecurity KSTs, a panel of 20 Subject Matter Experts (SMEs) was engaged to evaluate and validate the proposed competency measures. The expert panel refined the cybersecurity scenarios and experimental procedures used in the commercial cyber range hands-on scenarios, ensuring alignment with DCWF. Our findings indicated that 46 of 47 fundamental cybersecurity KSTs were validated by the SMEs as essential components of the FCCI. Consensus levels of 85–90% confirmed strong expert support for incorporating GenAI (e.g., large language models such as ChatGPT) as a teammate or decision-support agent in these controlled experiments. The validated scenarios and experiments pave the way for future research on assessing cybersecurity competencies in commercial cyber range platforms with and without GenAI support (e.g., large language models such as ChatGPT). By establishing the baseline for competency assessment in this research, the SMEs’ feedback contributed to advancing cybersecurity workforce development and provided critical insights for integrating GenAI into collaborative cybersecurity human-GenAI teaming operations. The validated FCCI provides a robust mechanism to evaluate both human and human–GenAI team performance within realistic cybersecurity scenarios, while providing the needed metrics to measure cybersecurity competencies quantitatively. While this study achieved strong consensus, like any other research, several limitations were observed, including a relatively small SME panel size (n=20) and the absence of empirical testing with users. Future research will employ hands-on cyber range experiments to measure the FCCI by comparing KSTs measured across human-only and human–GenAI teams. Ultimately, this research advances cybersecurity workforce development by establishing a validated foundation for a quantitative assessment of cybersecurity competencies based on DCWF necessary for effective collaboration between humans and GenAI in defending against complex and evolving cyber threats.
Comments
Extended abstract only for the proceedings and will submit the revised full paper to JCERP.