Tuesday, November 28, 2023
Terry Tush | Director of Marketing & Communications | 405-744-2703 | firstname.lastname@example.org
ChatGPT and other forms of artificial intelligence have been the talk of technology
circles, and it’s easy to see why.
Just by scraping the internet, ChatGPT and AI programs like it have passed the Graduate
Management Admission Test, aced the bar exam, written novels in the blink of an eye
and even hosted podcasts thanks to their ability to mimic the human voice.
So, it was only natural that accounting faculty nationwide collectively wondered about
AI and its application in their field. Will AI programs drastically change accounting,
they wondered? Will AI change the way they teach accounting? What does the existence
of this software mean for today’s students as they enter the workforce? Are AI programs
like ChatGPT already ahead of today’s accounting students?
There was only one way to find out. It was time to put ChatGPT to the test, literally.
Oklahoma State University Spears School of Business accounting professors Brad Lawson
and Bryan Brockbank joined forces with accounting faculty from across the globe to
pit their current students against ChatGPT version 3.5, which was introduced to the
world in November 2022.
The study represented one of the largest in the history of accounting research. Brigham
Young University’s Dr. David Wood led the research project, and he partnered with
327 faculty members at 186 institutions across 14 countries on the study. The entire
process took only about three months — lightning pace for accounting research — simply
because the technology and crowdsourced approach to data collection made the breakneck
Faculty in the study were asked to feed exam questions into ChatGPT that their accounting
students had been asked the previous semester. Then, record whether the chatbot got
the answer correct, partially correct or completely wrong, and upload the results
to a database.
In all, 28,085 questions were loaded into ChatGPT. Spears Business faculty didn’t
know what to expect as they clicked the chatbot’s send button.
“Based on the news stories I had heard, I was afraid that ChatGPT was going to do
pretty well,” said Dr. Lawson, who teaches accounting information systems and data
analytics. “I had heard that it was passing the bar exam and doing well on master’s
level courses. So, my fear was that it was going to actually do pretty well on our
How did it turn out? The short answer is that the accounting students came out on
top, for now, but it comes with an asterisk and a lesson about the future of accounting
Overall, the flesh and blood students posted an average of 76.7% compared to ChatGPT’s
47.4%, but the chatbot’s numbers jump to 56.5% if partial credit is factored in. ChatGPT
also performed better than the average student on 15.8% of the assessments.
The study also identified variation across accounting topics. ChatGPT performed better
on auditing and accounting information system assessments compared to other accounting
topics, like tax, financial and managerial accounting.
ChatGPT version 3.5 struggled with short-answer questions, and in fact, would sometimes
produce false and entirely fabricated answers and references. It also struggled with
questions that involved math calculations because ChatGPT is a language-based program.
Many times, the software didn’t even recognize that it was being asked a math problem.
In essence, it was trying to use a dictionary to solve complex calculations. This
could explain the variation across accounting topics because auditing and information
systems often have fewer mathematical questions than the other topics.
The latest iteration of the program, ChatGPT version 4.0, has no such problem. It
was released on March 14, 2023, which is ironic because the crowdsourced paper “The
ChatGPT Artificial Intelligence Chatbot: How Well Does It Answer Accounting Assessment
Questions?” was accepted into Issues in Accounting Education on the exact same day.
Version 4.0 of the chatbot includes the ability to do complex math calculations that
version 3.5 couldn’t comprehend.
Within hours of its release, ChatGPT version 4.0 passed every major accounting certification
exam, including the Certified Public Accountant, Certified Internal Auditor and the
Certified Management Accountant. The program went from flunking college exams in November
to showcasing the same knowledge as a certified professional in just five months.
If accounting faculty weren’t convinced by the research project’s results, the latest
iteration of ChatGPT put their doubts to rest. It was officially time to adjust curriculum
to teach students how to adapt to changing technology.
“That’s the approach we’ve taken at OSU,” Lawson said. “Yes, we’re going to teach
them Excel. They’re going to learn Tableau. They’re going to learn certain technologies
that are the expectation in the profession, but it’s more about teaching the skills
and the fundamentals they need in order to adapt and change as the technology changes
because that’s going to serve them better five years from now.”
Initial concerns around ChatGPT and academia were about cheating. Faculty were concerned
about students using the platform to write term papers and complete projects in a
21st century version of plagiarizing. But the thought has shifted to the inevitable
future. These types of programs aren’t going away, and in fact, they’re probably going
to change the way industries operate.
Public accounting firms have invested billions of dollars into AI research and advanced
data analytic technologies because they see the possibilities on the horizon. Academics
across the country have also begun to ask how they can use AI in the classroom to
prepare students for that reality.
One way is to incorporate ChatGPT into tests and then ask students to expound on the
chatbot’s answer or explain why it’s correct or incorrect. After all, even the latest
iterations of the program are far from perfect, and the modern workforce will need
to be able to tell when an AI program is going down the wrong path.
“It’s a different level of thinking,” Dr. Brockbank said. “I think that’s where the
future with ChatGPT or AI is going to be. We have to teach students to think more
critically about what answers ChatGPT is giving you because when ChatGPT’s answers
come out, the wording sounds very authoritative. It says this is the correct answer
and this is why, even when it was wrong. From the student perspective, if you’re using
something like this and you don’t know, you would believe it 100% — this is the right
Instilling the critical thinking skills needed to navigate the reality of AI alongside
the basics of accounting is essential to creating a workforce that can use professional
judgment to make the best decisions for their company.
Thanks to Brockbank and Lawson, Spears Business students will have that and more.
Along with the other faculty researchers involved in the study, they see that the
field of accounting isn’t going anywhere, and neither is AI.
“AI is not going to replace accountants,” Wood said. “Accountants using AI are going
to replace accountants who are not using AI. I think in the immediate future that’s
going to be the case. It just makes people so much more productive.”
Story by: Stephen Howard | Discover@Spears Magazine
Photo by: Devin Flores