In recent weeks, there has been a rising tide of skepticism about the effectiveness of AI coding assistants, especially those built on top of OpenAI’s models. The same author conducted a systematic test to see if the performance of these tools has degraded over time. I ran the same ten queries across different iterations of ChatGPT and GPT-4 (and the newly-released GPT-5). The intention was to correct a coding mistake. Overall, the results showed a stark contrast in performance between the two models. The news raised new concerns over the quality and reliability of AI-assisted coding.
The author nonetheless notes a potentially promising trend from recent months. Six months later, the majority of AI coding companions miss the mark in significantly assorted ways on developers’ anticipations. This article will dive into what we learned from the fated tests. It further looks at what these developments mean for the future of AI in programming.
Testing Methodology
To evaluate the performance of AI coding assistants, the author ran a simple but powerful test. The goal was to figure out whether these tools were actually becoming worse at the task of writing code in the first place. The author was able to elicit this error message from nine different versions of ChatGPT. They specifically zeroed in on GPT-4 and GPT-5, prompting each model for a finished code solution with no additional explanation.
The process was methodical. The author specified that only functional code was acceptable, eliminating any explanations or side comments that could distract from the task at hand. This strategy was designed to ensure that the models’ effectiveness could be isolated to their ability to provide direct coding solutions.
Performance of GPT-4
The outcomes from GPT-4 showed an overall better performance. In ten attempts, it was able to give a helpful response each time. However, there were notable shortcomings. In half of those six examples, GPT-4 actually tried to run the code. It added an exception that would either raise an error or populate the new column with an error message when the specified column didn’t exist. This methodology reflects the AI models’ proclivity to approach coding problems with a level of conservatism. Unfortunately, this overzealous caution can add layers of complication for developers that are wholly unnecessary.
In one instance, GPT-4 just repeated the prompt code rather than providing a better answer. This redundancy represents a worrying stop-gap of its crisis management capacity. For nine of the ten test cases, GPT-4 explicitly enumerated a list of columns in the dataframe. In the process, however, it made the case that the missing column should have been in the dataset. This behavior clearly shows that GPT-4 had a historic tendency to focus on natural language flow first and accurate code solution second.
In those three cases, GPT-4 ignored the explicit instruction to only output code. Rather than provide an apology, the paper decided to report on the anticipated absence of the column. These types of deviations from the outputs requested can alienate users looking for a simple, easy answer.
Performance of GPT-5
GPT-5 was a clear leap forward in reliability and accuracy. Or that it really nailed the sensor fusion solution that worked in every test scenario run all the time. The model’s approach to culture change was remarkably efficient. Instead, it used the underlying indexing of each row, incrementing by 1 to create a new column—this is an example of a much better problem-solving alternative.
If this is the consistency we can expect from GPT-5, it suggests that even radical improvements in AI technology will lead to tangible benefits for developers. With GPT-5, OpenAI raises the standard for AI coding assistants even further. It provides full-fledged, working code examples without a bunch of barking and hand-wavy mistakes.
This important distinction between the two models begs the question on how AI capabilities are iterating on a larger scale. As you can tell, GPT-4 holds some amazing potential. Despite its strengths, its limitations are keeping it behind the curve of user expectations and the potential of generative AI technology.
Anecdotal Evidence of Declining Performance
The author’s anecdata on production flow corroborates that indeed AI-assisted coding is trending toward less efficient delivery as we go back in time. This has led to tasks that used to take about five hours now take seven or eight hours because of these stresses. The tradeoff is that developers would likely have to invest additional time polishing or otherwise fixing AI-generated code. They can’t expect to turn to those assistants and get a quick fix anymore.
In addition, the anecdotal evidence suggests that many users have been experiencing the same problems with AI coding assistants. Assuming various core models, we may hit a performance plateau sometime around 2025. This impending loss is sending up alarm bells among developers that depend on these tools to improve their own productivity to realize it.
Despite initial optimism surrounding AI’s potential to revolutionize coding practices, the recent experiences reported by users indicate a pressing need for improvements in AI technology. As developers increasingly depend on these tools for assistance, addressing these performance issues will be critical for maintaining their utility in software development.
