In a new study, ChatGPT tackled a classical ancient Greek math problem, revealing both its prowess for improvisation and its digital shortfalls. The experiment was conducted by Dr. Nadav Marco, visiting scholar at the University of Cambridge. He teamed up with Professor Andreas Stylianides to examine the way AI approaches mathematical problems that have stumped scholars for hundreds of years. The problem posed in the challenge was a form of the so-called “doubling the square” problem, a mathematical challenge proposed by Plato circa 385 BCE.
Participants were hoping to see ChatGPT quickly reach into its deep training library to remember Socrates’ classic answer to this challenge. When tasked with the simple task of doubling the size of a triangle, ChatGPT took some time before suggesting the well-known solution. Instead, it was slow to react. At times, it felt like it was winging it. This approach emulated that trial and error process of a student learning the material from scratch when faced with difficult pre-algebra concepts.
The Challenge of Doubling the Square
The “doubling the square” problem is an old geometric challenge. Specifically, it dares you to create a square that has twice the area of an existing square. Plato would give this allegory of mathematics in his dialogues, stressing its importance and utility in grasping fundamental truths on a deeper level. When ChatGPT faced this challenge, it was subjected to various prompts from researchers, including “I want us to explore this problem together” and “Tell me the answer.”
Early on, ChatGPT returned answers without the accuracy and specificity one would hope from a specialized solution. Yet it made egregiously human-like mistakes in its efforts to solve the problem, showing a surprising amount of second-guessing. Such behavior not only called into question the nature of AI’s mathematical ‘knowledge’ but what users may assume about the AI’s output when used in a classroom environment.
Dr. Marco further elaborated on this use case by emphasizing that ChatGPT behaves as the ultimate student. He likened its trial-and-error process to that of a math student who is still learning his multiplication tables. The AI’s failure to produce a timely and accurate response underscored the complexity of machine learning and knowledge search.
Human-like Errors and Geometrical Alternatives
As the research wore on, ChatGPT was told that it was failing to write an “elegant and exact” response. With this knowledge, it through trial and error produced a purely geometric second best solution. This moment illustrated a key aspect of its design: while it can generate responses based on existing knowledge, it can adapt its reasoning when prompted.
These qualitatively human-like errors sparked major debate over the possibility of learning from error. In discussing these types of mistakes, researchers found that these could be valuable teaching opportunities when incorporated into pedagogical approaches. If we are watching AI struggle with a problem, we can use this insight to craft lessons and curricula that better prepare our students for those same faults.
These results highlight the need to cultivate expertise in assessing AI-produced proofs. These days, artificial intelligence seems like it’s everywhere—especially in education. Students need to be trained to critically evaluate the outputs since AI will be the pervasive undercurrent of this new ecosystem.
Implications for Mathematics Education
In a recent research study, Dr. Marco and Professor Stylianides put ChatGPT to the test. Often, they speculated about the broader implications for mathematics education. Their results show that AI tools can significantly support learning. They are equally quick to insist on the need for human oversight and contextual understanding.
Educators can take advantage of ChatGPT’s weaknesses by operating within its Zone of Proximal Development (ZPD). Such an approach opens the door to more meaningful learning experiences. This method truly engages students and immerses them in the process of how we define solid mathematical practices. Simultaneously, it trains them to evaluate and improve upon AI-generated solutions.
As educational institutions continue to incorporate technology into their curricula, integrating skills related to AI evaluation will become increasingly important. Understanding how AI interprets and solves problems can empower students to think critically about mathematical reasoning and develop their problem-solving skills.