Department
Department of Information Systems and Security
Document Type
Article
Publication Date
2-5-2025
Embargo Period
4-5-2025
Abstract
Large language models such as ChatGPT provide efficient access to a wealth of information. However, there are significant questions regarding the depth and quality of knowledge in any one domain. This paper focuses specifically on the information systems (IS) field and assesses ChatGPT’s knowledge. To analyze the extent and quality of information systems knowledge derived from queries to ChatGPT, we used over 3,000 queries from a broad range of exam and quiz questions. These queries were obtained from university courses and professional information system certification exams. The query topics are based on a framework for information systems education, with queries gathered from multiple faculty members at different universities. The results of querying ChatGPT with these questions find that ChatGPT can answer 65% to 85% of information systems queries successfully, across each of the areas of the framework. ChatGPT performed well on essay, true/false and multiple choice questions, with no statistical difference between its success rate on multiple choice and true/false questions. We conclude that ChatGPT tends to perform approximately at the level of an average student, but may not perform at a level sufficient to pass certain professional exams. ChatGPT’s knowledge is very broad, covering virtually all areas of the information systems knowledge we identified, but not at an expert level.
Journal Title
Communications of the Association for Information Systems
Journal ISSN
1529-3181
Volume
56
Issue
1
First Page
123
Last Page
143
Digital Object Identifier (DOI)
n/a
Included in
Management Information Systems Commons, Scholarship of Teaching and Learning Commons, Science and Mathematics Education Commons, Technology and Innovation Commons
