Strengthening Cybersecurity Research with Applied Mathematics and Data Science Approaches

The scope of cybersecurity extends far beyond the theoretical realm, as cybersecurity practitioners must secure critical infrastructure like the Internet, transportation networks, and power grids. However, cybersecurity research often lacks rigorous mathematical foundations, instead relying on informal assumptions, context-specific simulations, and untested models. As cybersecurity threats grow and change, this lack of rigor raises concerns about the reliability and confidence of established methods. 

During the Third Joint SIAM/CAIMS Annual Meetings, which are taking place this week in Montréal, Québec, Canada, Pablo Moriano of Oak Ridge National Laboratory delivered a minisymposium presentation on the ways in which tools from modern science—particularly in applied mathematics and data science—can improve the trustworthiness and impact of cybersecurity research. “It’s not enough to seem plausible,” Moriano said. “The work also needs to be applicable, transferable, and reproducible.” 

Overarching philosophies on how science is carried out have evolved over time, but the discipline of cybersecurity still often follows outdated modes of thinking. “Many of us have seen a lack of reproducibility,” Moriano said. “What if the problem isn’t the field at all, but the assumptions about what science is?” The cybersecurity community has yet to embrace modern pragmatism, a currently widespread school of thought which asserts that good science is an iterative process that should follow certain steps: using observations to make inductions and find models of behavior, testing via comparison with natural phenomena, and following a continuous cycle of refinement to improve scientific claims (see Figure 1). “It’s not only about doing it once or twice,” Moriano said. “There’s going to be a lot of iteration and refinement.”

Modern science perspectives can help address some common critiques about cybersecurity (see Figure 2). Making structured observations, seeking mechanisms rather than natural laws, and embracing various forms of reproducibility are all aspects of modern pragmatism that cybersecurity could benefit from adopting. “The most important idea that I want you to get from this talk is that science is a process, not a product,” Moriano said.

When the field of biomedicine faced a reproducibility crisis, the community responded by jumpstarting the discipline of meta-research: the study of how science is performed. There is much that cybersecurity researchers can gain from adopting meta-research ideas surrounding (i) methods, in terms of study design and statistical rigor; (ii) reporting, in terms of transparency and open protocols; (iii) reproducibility, in terms of access to data, code, and methods; (iv) evaluation, in terms of peer review and funding criteria; and (v) incentives, in terms of aligning rewards to robustness versus novelty. A 2023 study applied the meta-research framework to review the focus areas of approximately 100 cybersecurity papers [1]. A total of 80 percent of these articles focused on methods, with a paltry five percent of the texts focusing on evaluation and incentives combined.

Moriano next presented a case study of adversarial machine learning: an interest area in cybersecurity that has seen explosive growth in recent years. A meta-research point of view raises a number of concerns when much of this work comes under scrutiny. Many papers use cherry-picked datasets and inconsistent evaluation metrics, also often failing to test real-world viability.

“What would it take for us to grow cybersecurity and make a stronger science?” Moriano asked. He provided several recommendations of observation, modeling, evaluation, and meta-research practices that could help develop a more robust science of cybersecurity (see Figure 3). For instance, observations should entail real-world measurements made with careful attention to detail, while models should have well-defined assumptions and constraints. Open sharing of benchmarks and reproducible pipelines could improve evaluation practices, as could incentives for robustness and generalizability. Furthermore, meta-research initiatives should continue to study how all of this work gets done.

Data scientists and applied mathematicians have much to offer in these pursuits. The discipline of data science brings tools in statistical validation, benchmarking, reproducibility, logging, and visualization, whereas applied mathematics can provide insights into formal systems, simulations, inference, and generalization.

To conclude, Moriano offered several concrete recommendations of ways in which cybersecurity researchers can immediately begin to improve their practices: 

• Treat models as tools, rather than abject truths
• Embrace structured observations and falsifiability
• Share data, code, and evaluation procedures
• Reward rigor, rather than simply seeking novelty.

Looking towards the future, Moriano hopes to see greater investments in infrastructure for reproducibility and more support for cybersecurity meta-research. Aligning funding priorities with these values and providing robust training for the future cybersecurity workforce on rigorous scientific principles will serve to further these goals. Applied mathematicians and data scientists can contribute significant insights to these discussions, and can find exciting opportunities to collaborate with cybersecurity experts to develop sophisticated tools to address modern cyber threats.

References
[1] Le Pochat, V., & Joosen, W. (2023). Analyzing cyber security research practices through a meta-research framework. In CSET ’23: Proceedings of the 16th cyber security experimentation and test workshop (pp. 64-74). Marina del Ray, CA: Association for Computing Machinery.
[2] Spring, J.M., Moore, T., & Pym, D. (2017). Practicing a science of security: A philosophy of science perspective. In NSPW '17: Proceedings of the 2017 new security paradigms workshop (pp. 1-18). Santa Cruz, CA: Association for Computing Machinery.