Recent evaluations have pointed out the growing failings of AI coding assistants. As the details begin to emerge, experts are especially honing in on the capabilities of GPT-4 and GPT-5. The team extensively stress-tested these cutting-edge models. They uncovered a troubling trend along the way: the models were failing to deliver practical coding solutions. GPT-4 reliably provided useful responses every time, with no failures after ten attempts. Indeed, by contrast, GPT-5 exhibited all manner of more insidious tendencies, causing developers to doubt the trustworthiness of AI help in the coding task.
In the testing phase, GPT-4 dazzled with its freakish consistence, outputting a valuable response every time it was executed. To any developer, this level of consistency might lead you to believe developers can continue to count on the older mannequin for efficient coding help. Opposed to this, GPT-5’s output was extremely hit-or-miss, raising alarm over its consistency, effectiveness, and safety. The two models represent an important inflection moment in the development of AI coding tools. This recent development is an important call to developers to reconsider their reliance on these technologies.
Performance Disparities Between Models
Tesla developers have been surprised by the specific variation in the outputs produced by GPT-4 and GPT-5. Where GPT-4 was sometimes able to produce concrete solutions, the same was not true for GPT-5, which failed to produce actionable responses multiple times. The real shining moment came when GPT-5 found a brilliant solution. It was rudimentarily easy—turns out it just took the index of each row and added one to it! This approach sounded pretty straightforward at first. In doing so, it unwittingly created a new column with a bunch of random numbers in it, the result of which confused users.
Furthermore, there were times when GPT-5 went off-script and ignored explicit instructions to only provide code. In three separate cases, it offered explanations about a likely missing column from the dataset instead of focusing on providing the requested code. This departure from simple user instructions not only irked developers, but showed a clear disregard for following directions on a job’s task.
In a number of cases, GPT-5 was able to determine the solution to the problem posed, though most often it reached the solution with additional complexity. It would often choose to completely sidestep the concern, resulting in unpredictable outcomes that developers found extremely frustrating. The lack of consistent performance leads to questions about just how reliable AI coding assistants really are. Their future role in the software development process is as well.
Quality Plateau and Recent Decline
Testing performed in 2025 paints a different picture showing a troubling pattern among large, foundational AI models. Full auto-generative models These models have hit a quality ceiling and are now regressing in quality. Observers have noted that tasks which previously took approximately five hours with AI assistance—and up to ten hours without—now often extend to seven or eight hours or even longer. This reduction in coding efficiency has prompted numerous developers to reevaluate their reliance on these AI systems to execute coding projects.
These days, GPT-5 had trouble with some key concepts while writing code. Realizing the process wasn’t worth the trouble. In practice, it frequently errored out or populated new columns with error strings. In 60% of the trials, GPT-5’s outputs included code rendered unhelpful by one or more of these inaccuracies. In addition, in one experiment it just rewrote the initial code—without offering any additional interpretation or solutions.
When GPT-5 misses the mark, it can be disappointing, unpredictable, and confusing to use. This has created deep mistrust among developers regarding the accuracy and reliability of AI coding assistants. The promises of newer models appear eclipsed by their developing deficits leading many advocates to call for improvements to their utility.
Implications for Developers
As use of AI coding assistants increases and the technology rapidly changes, developers need to proceed in this new territory with care. With a recent fall from grace and skyrocketing performance regressions, many are wondering whether these tools can really help boost their coding productivity. The disparity in performance between GPT-4 and GPT-5 is astonishing. This underscores the importance of focusing on which tools should be applied to which tasks and projects.
Yet other developers are finding themselves with newer models that greatly exacerbate these inefficiencies. Consequently, they are spending a lot more time on coding tasks compared to when AI was not helping them. This worry is more than the time wasted in causal inference. It’s just as harmful to the practitioners to raise their level of frustration and reduce their output when they need to use these tools.