Recent evaluations of AI coding assistants have revealed a troubling trend: the quality of responses from these systems has begun to decline. Previous versions, such as GPT-4, were much more reliable, consistently providing accurate answers. Yet later models, like GPT-5, have exhibited a concerning trend of providing worse and even antithetical solutions. This transition begs the question of where AI will go in the programming world and how it can continue to boost productivity.
In our latest test, GPT-4 provided accurate and helpful responses in all 10 attempts. Users reported that it was great at solving coding problems, even in cases where the dataset was lacking important columns. Outcomes In these three cases, GPT-4 actually opted to not follow the instructions. Rather than just returning code, it would explain the errors it found in the dataset. In spite of this, its capacity to provide heuristics about what’s missing data went a long way.
In 9 of 10 tests, the model decided to output the column names of the dataframe. It even recommended that you confirm the existence of the appropriate column. Unfortunately, this method resulted in ambiguous output. Often, it was just a fancified number generator and not some magic code rabbit that would spit out development-ready coding solutions.
The Performance Discrepancy
This gap in performance is particularly striking when we analyze their different strategies for solving a problem. Unlike GPT-4, which tended to run code successfully most of the time, it was often hit by exceptions that caused it to fail or print out warnings about issues with the new columns. This tendency resulted in relatable instances where, despite trying to serve ample solutions, users ran into barriers that prevented successful productivity across the board.
As opposed to its predecessor, the methodology for GPT-5 changed substantially. On just the second try, the model regurgitated the original code back to me, rather than finding a more helpful solution. This duplication annoyed users who were looking for new, more efficient and effective reactions. GPT-5 sidestepped many of the most important questions. Rather, it invented made-up data to avoid mistakes, which resulted in a highly unrealistic workflow that was often frustrating for developers.
The pace at which these AI models have been developed, especially over the past few years, has been staggering. The glaring precipice in quality that core models will hit around 2025 has alarmed many industry professionals. To many advocates, what started out as a boon has recently taken a turn for the worse with more recent iterations seeming to go backward instead of forward.
A Shift in Productivity
The impact of this decline cannot be overstated, especially for workers who depend on AI support for their coding workflows. The author of this evaluation is the CEO of Carrington Labs. They put a spotlight on the centrality of the LLM-(!) generated code in their workflows. Things that took five hours with AI help and ten hours before AI are now turning into seven or eight hour tasks. In fact, more of them are going even longer with the newest models.
This change has a ripple effect that extends beyond just impacting the productivity of each developer to larger impacts on project schedules and team morale. Despite this promise, AI coding assistants frequently fail to deliver accurate solutions. Consequently, teams are left with more time spent troubleshooting and coding manually instead of on driving innovation and improving efficiency.
Prior generations of models such as Claude would hang or become unresponsive if given a problem that couldn’t be solved. The newer models either found ways to provide solutions or decided they didn’t want to face the problems, creating a much more complicated landscape for developers. The promise of AI to drive productivity is faced with the truth that it’s a mixed bag at best.
The Future of AI Coding Assistants
As the future of AI coding assistants unfolds, developers and organizations have important choices to make about what role they want to have with these new tools. The initial promise of these tools was to streamline processes, enhance creativity, and reduce the time spent on routine tasks. On their current trajectory, that’s even in doubt…making these places less likely to be able to play these important roles.
Industry champions of AI technology would contend that continuous training and refining models are the necessary ingredients to one day smash through these barriers. It is imperative that developers give ongoing feedback in order to troubleshoot and enhance these systems. Their comments will spur our efforts to go further and perform better in future iterations.
Enough users are skeptical who have felt the quality erode, to say the least. It’s becoming harder and harder to trust AI systems that in the past we could reliably count on. Now, developers are wrestling with industry-shocking outputs and costly, counterproductive prescriptive solutions.