AI chatbots and cybersecurity: What do they tell the public?

Abstract

This study evaluates the performance of five large language model chatbots - ChatGPT, Copilot, Gemini, Claude, and Cohere - on topics related to cybersecurity, healthcare, and the environment. The chatbots were evaluated by asking five specific questions, and their responses were analyzed to determine how well they aligned with the cybersecurity principles outlined in the Paris Call. To assess the semantic similarity of their responses, 384-dimensional sentence embeddings from Hugging Face (HuggingFace.com) were used to calculate cosine distances, offering a quantitative measure of their alignment with the Paris Call principles. Repeated measures of analysis of variance (ANOVA) revealed no significant differences in how frequently the chatbots applied the nine Paris Call principles to individual questions, with similar application rates across the chatbots. However, a separate ANOVA across all five questions identified significant differences (p = 0.011) in the average use of these principles, suggesting that the chatbots likely rely on different datasets and did not consistently apply the principles across all questions. The study also found errors of omission, where certain key principles were left out of some responses. For example, several chatbots failed to mention critical elements, e.g. protecting the integrity of supply chains or ensuring accountability in technology use, highlighting gaps in their cybersecurity coverage. As a result, users may need to query multiple chatbots to gain comprehensive insights on these topics.

Keywords

Large language models / Artificial intelligence / Paris call for trust and security in cyberspace

Acknowledgments

None.

Funding

None.

Conflict of interest

The authors declare no conflicts of interest.

Author contributions

Conceptualization: Gloria Wu, Kayla Draves-Hau

Data curation: Milan Del Buono, Adrial Wong

Formal analysis: Milan Del Buono, Adrial Wong

Investigation: Gloria Wu, Kayla Draves-Hau, Adrial Wong

Methodology: Gloria Wu, Kayla Draves-Hau

Writing-original draft: Gloria Wu, Kayla Draves-Hau

Writing-review & editing: Gloria Wu, Adrial Wong, Milan Del Buono, Mary Nguyen, Hasini Namala

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data

Data are available from the corresponding author upon reasonable request.

References

[1] Alanazi A.T., & Alanazi A. (2023). Clinicians’ perspectives on healthcare cybersecurity and cyber threats. Cureus, 15(10):e47026. https://doi.org/10.7759/cureus.47026

[2] AL-Hawamleh A.M. (2023). Predictions of cybersecurity experts on future cyber-attacks and related cybersecurity measures. International Journal of Advanced Computer Science and Applications, 14(2). https://doi.org/10.14569/IJACSA.2023.0140292

[3] Alshahrani H., Islam N., Syed D., Sulaiman A., Al Reshan M.S., Rajab K., et al. (2023). Sustainability in blockchain: A systematic literature review on scalability and power consumption issues. Energies, 16(3):1510. https://doi.org/10.3390/en16031510

[4] Arsenault A.C., Kreps S.E., Snider K.L., & Canetti D. (2024). Cyber scares and prophylactic policies: Cross-national evidence on the effect of cyberattacks on public support for surveillance. Journal of Peace Research, 61(3):413-428. https://doi.org/10.1177/00223433241233960

[5] Badrouchi F., Aymond A., Haerinia M., Badrouchi S., Selvaraj D.F., Tavakolian K., et al. (2020). Cybersecurity vulnerabilities in biomedical devices:A hierarchical layered framework. Internet of Things Use Cases for the Healthcare Industry. Cham: Springer, p. 157-184. https://doi.org/10.1007/978-3-030-37526-3_7

[6] Bell N., & Mbaziira A. (2023). Exploring a Diplomatic System of Cooperation in the Cyber Space through a Proposed Cyber Diplomacy Cooperation Framework. In: International Conference on Cybersecurity and Cybercrime. Vol. 10, p.7-11. https://doi.org/10.19107/CYBERCON.2023.01

[7] Biswas S.S. (2023). Role of chat GPT in public health. Annals of Biomedical Engineering, 51(5):868-869. https://doi.org/10.1007/s10439-023-03172-7

[8] Cartwright A.J. (2023). The elephant in the room: Cybersecurity in healthcare. Journal of Clinical Monitoring and Computing, 37(5):1123-1132. https://doi.org/10.1007/s10877-023-01013-5

[9] Choudhury A., & Shamszare H. (2023). Investigating the impact of user trust on the adoption and use of ChatGPT: Survey analysis. Journal of Medical Internet Research, 25:e47184. https://doi.org/10.2196/47184

[10] Chouhan A.S., Qaseem M.S., Basheer Q.M.A., & Mehdia A. (2023). Blockchain based EHR system architecture and the need of blockchain in healthcare. Materials Today: Proceedings, 80:2064-2070. https://doi.org/10.1016/j.matpr.2021.06.114

[11] Dameff C., Tully J., Chan T.C., Castillo E.M., Savage S., Maysent P., et al. (2023). Ransomware attack associated with disruptions at adjacent emergency departments in the US. JAMA Network Open, 6(5):e2312270. https://doi.org/10.1001/jamanetworkopen.2023.12270

[12] De Kok T. (2024). ChatGPT for Textual Analysis? How to Use Generative LLMs in Accounting Research. SSRN. https://doi.org/10.2139/ssrn.4429658

[13] Derecho K.C., Cafino R., Aquino-Cafino S.L., Isla A. Jr., Esencia J.A., Lactuan N.J., et al. (2024). Technology adoption of electronic medical records in developing economies: A systematic review on physicians’ perspective. Digital Health, 10:20552076231224605. https://doi.org/10.1177/20552076231224605

[14] Dyer O.(2024). US hospitals face collapse as cyberattack on UnitedHealth cuts revenue streams. BMJ, 384:q686. https://doi.org/10.1136/bmj.q686

[15] Eppler M., Ganjavi C., Ramacciotti L.S., Piazza P., Rodler S., Checcucci E., et al. (2024). Awareness and use of ChatGPT and large language models: A prospective cross-sectional global survey in urology. European Urology, 85(2):146-153. https://doi.org/10.1016/j.eururo.2023.10.014

[16] Fernando Y., & Saravannan R. (2021). Blockchain technology: Energy efficiency and ethical compliance. Journal of Governance and Integrity, 4(2):88-95. https://doi.org/10.15282/jgi.4.2.2021.5872

[17] Firat M. (2023). How Chat GPT can Transform Autodidactic Experiences and Open Education? OSF Preprints. https://doi.org/10.31219/osf.io/9ge8m

[18] Fox S., & Duggan M. (2013). Health online 2013, p. 1-55. Available from: https://www.pewresearch.org/internet/2013/01/15/health-online-2013 [Last accessed on 2024 Jul 08].

[19] George A.S., & George A.H. (2023). The emergence of cybersecurity medicine: Protecting implanted devices from cyber threats. Partners Universal Innovative Research Publication, 1(2):93-111. https://doi.org/10.5281/zenodo.10206563

[20] Hugging Face. (n. d.). Available from: https://huggingface.co/ sentence-transformers/all-minilm-l6-v2 [Last accessed on 2024 Jul 08].

[21] ISO/IEC 27001:2022. ISO. (2022). https://www.iso.org/standard/27001 [Last accessed on 2024 Jul 08].

[22] Jakubek K. (2022). AMA Adopts New Policy Declaring Climate Change a Public Health Crisis. American Medical Association. Available from: https://www.ama-assn.org/press-center/press-releases/ama-adopts-new-policy-declaring-climate-change-public-health-crisis [Last accessed on 2024 Jul 08].

[23] Javaid M., Haleem A., Singh R.P., & Suman R. (2023). Towards insighting cybersecurity for healthcare domains: A comprehensive review of recent practices and trends. Cyber Security and Applications, 1:100016. https://doi.org/10.1016/j.csa.2023.100016

[24] Kelly B., Quinn C., Lawlor A., Ronan K., & Burrell J. (2023) Cybersecurity in healthcare. In: Trends of Artificial Intelligence and Big Data for E-Health. Germany: Springer Nature, p. 213-231. https://doi.org/10.1007/978-3-031-11199-0

[25] Kormiltsyn A., Udokwu C., Dwivedi V., Norta A., & Nisar S. (2023). Privacy-conflict resolution for integrating personal and electronic health records in blockchain-based systems. Blockchain in Healthcare Today, 6(2):6. https://doi.org/10.30953/bhty.v6.276

[26] Machal M.L. (2023). An overview about connected medical devices and their risks. Studies in Health Technology and Informatics, 305:119-122. https://doi.org/10.3233/SHTI230438

[27] Minnaar A., & Herbig F.J.W. (2021). Cyberattacks and the cybercrime threat of ransomware to hospitals and healthcare services during the COVID-19 pandemic. Acta Criminologica: African Journal of Criminology and Victimology, 34(3). https://hdl.handle.net/10520/ejc-crim_v34_n3_a10

[28] Ortiz A. (2024). AT and T Resets Millions of Passcodes after Customer Records are Leaked. The New York Times. Available from: https://www.nytimes.com/2024/03/30/business/att-passcodes-reset-data-breach.html [Last accessed on 2024 Jul 08].

[29] Pascoe C.,Quinn S., & Scarfone K. (2024). The NIST Cybersecurity Framework (CSF) 2.0. NIST Cybersecurity White Papers (CSWP). https://doi.org/10.6028/NIST.CSWP.29

[30] Pesl R.D., Stötzner M., Georgievski I., & Aiello M., (2023). Uncovering LLMs for Service-composition:Challenges and Opportunities. In: International Conference on Service- Oriented Computing, p.39-48. https://doi.org/10.18419/darus-3767

[31] Ratasha M.A.B., Ismail M.S.B., Aziz A.S.A., Williams G., & Rees E. (2024). Towards Net Zero-better Decision Making for Enterprise Level Green House gas (GHG) Emission Management and Accounting. In: InternationalPetroleum Technology Conference,D021S052R005. https://doi.org/10.2523/IPTC-23574-MS

[32] Reddy J., Elsayed N., ElSayed Z., & Ozer M. (2023). A review on data breaches in healthcare security systems. International Journal of Computer Applications, 184(45):1-4. https://doi.org/10.5120/ijca2023922333

[33] Roumeliotis K.I., & Tselikas N.D. (2023). Chatgpt and open-ai models: A preliminary review. Future Internet, 15(6):192. https://doi.org/10.3390/fi15060192

[34] Sallam M, Salim N.A., Barakat M., & Al-Tammemi A. (2023). ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J, 3(1). https://doi.org/10.52225/narra.v3i1.103

[35] Shahsavar Y., & Choudhury A. (2023). User intentions to use ChatGPT for self-diagnosis and health-related purposes: Cross-sectional survey study. JMIR Human Factors, 10(1):e47564. https://doi.org/10.2196/47564

[36] Shen C., & Wang Y. (2023). Concerned or apathetic? Exploring online public opinions on climate change from 2008 to 2019: A comparative study between China and other G20 countries. Journal of Environmental Management, 332:117376. https://doi.org/10.1016/j.jenvman.2023.117376

[37] Shi X., Xiao H., Liu W., Lackner K.S., & Stocker T.F. (2023). Confronting the carbon-footprint challenge of blockchain. Environmental Science and Technology, 57(3):1403-1410. https://doi.org/10.1021/acs.est.2c05165

[38] Shull A., Boysen A., Holland B., Hathaway M., Fay R., Desai N., et al. (2019). Governing Cyberspace During a Crisis in Trust. Centre for International Governance Innovation. Available from: https://www.cigionline.org/publications/governing-cyberspace-during-crisis-trust [Last accessed on 2024 Jul 08].

[39] Stergiopoulos G., Kotzanikolaou P., Konstantinou C., & Tsoukalis A. (2023). Process-aware attacks on medication control of type-I diabetics using infusion pumps. IEEE Systems Journal, 17(2):1831-1842. https://doi.org/10.1109/JSYST.2023.3236690

[40] Tangadulrat P., Sono S., & Tangtrakulwanich B. (2023) Using ChatGPT for clinical practice and medical education: Cross-sectional survey of medical students and physicians perceptions. JMIR Medical Education. 9(1):e50658. https://doi.org/10.2196/50658

[41] Temsah M.H., Aljamaan F., Malki K.H., Alhasan K., Altamimi I., Aljarbou R., et al. (2023). Chatgpt and the future of digital health: A study on healthcare workers perceptions and expectations. Healthcare, 11(13):1812. https://doi.org/10.3390/healthcare11131812

[42] Terán M., & Noelia H. (2023). Environmental sustainability and ICT: A look at impact factors. LUTPub. Available from: https://lutpub.lut.fi/handle/10024/165468 [Last accessed on 2024 Jul 08].

[43] The 9 Principles. (2018) Paris Call. Available from: https://pariscall.international/en/principles [Last accessed on 2024 Jul 08].

[44] Triplett W.J. (2024). Cybersecurity vulnerabilities in healthcare: A threat to patient security. Cybersecurity and Innovative Technology Journal, 2(1):15-25. https://doi.org/10.53889/citj.v2i1.333

[45] Van Boven L.S., Kusters R.W., Tin D., Van Osch F.H.M., De Cauwer H., Ketelings L., et al. (2024). Hacking acute care: A qualitative study on the health care impacts of ransomware attacks against hospitals. Annals of Emergency Medicine, 83(1):46-56. https://doi.org/10.1016/j.annemergmed.2023.04.025

[46] Wasserman L., & Wasserman Y. (2022). Hospital cybersecurity risks and gaps: Review (for the non-cyber professional). Frontiers of Digital Health, 4:862221. https://doi.org/10.3389/fdgth.2022.862221