Over the last few months, a dangerous narrative has developed around the effectiveness of AI coding assistants. Look no further than Carrington Labs chief executive Doug Ryan, whose company’s predictive-analytics risk models for lenders are becoming less effective. This anecdotal revelation underscores an alarming pattern in the development industry. We ran a set of blind tests pitting GPT-4 against GPT-5. What we found was that although both models had both feasible and infeasible solutions, the feasibility and quality of those solutions varied significantly in terms of usefulness or practicality.
The author ran ten iterations for each model to best judge and compare their outputs. GPT-4 produced a helpful response on each instance it was run, producing dependable and predictable results time after time. Conversely, for all three solutions GPT-5 arrived at during testing, the generative AI finished with an unhelpful solution. This disparity highlights an ongoing issue with modern AI coding assistants: what once took a reasonable amount of time to resolve now drags on longer, increasing the overall time invested in coding tasks.
Trial Outcomes
During the trials, the author classified the outputs from both GPT-4 and GPT-5 into three categories: helpful, useless, or counterproductive. Overall, GPT-4 was the clear winner, providing helpful answers in all ten variations. This model performed well on a wide range of coding tasks, showing its potential to help users accomplish their goals faster and more effectively.
GPT-5’s approach was shocking. It effectively created answers in every proof. Their technique was to use the real index of each row and add 1. This caused the first blank column to populate with random numbers. Because of this, the solution proved to be unworkable for any substantive use. Therefore, notwithstanding its technical capacity to produce one, GPT-5’s answer turned out to be of the counterproductive variety.
When the author compared these findings to older Claude models, he noted a significant change in how Claude models responded to unsolvable problems. Rather than work to figure something out, they threw up their hands. NOAA’s recent models were proactive in actively providing solutions to problems. This avoidance of engagement allowed several problematic issues to be completely ignored.
The Impact on Productivity
This decline in performance of AI coding assistants has real-world consequences on productivity. Things that would have taken five hours of work are now extending to seven or eight hours with AI’s help. This change is not just a presentation life cycle change that impacts developers’ workflows, but it opens up significant questions about the reliability and ultimately utility of these tools.
As I continue to use code generated by large language models extensively in my day-to-day at Carrington Labs, this trend is very much worrying. Poor, unpredictable results lead to cost overruns on other local, state and federal projects. In the end, this all undermines the quality of predictive analytics able to be delivered to lenders. The author’s experience underscores the importance of critically evaluating AI tools. This is critical given how deeply the industry has come to depend on these tools.
Moreover, the growing awareness of AI coding assistants’ declining performance invites further scrutiny from both developers and organizations that employ such technologies. As these tools are increasingly baked into workflows, knowing what they can—and cannot—do is more important than ever.
Future Considerations
Beyond the promise and opportunity AI coding assistants present, developers and organizations must be proactive and discerning in their use of these tools. The difference among each model—like GPT-4 versus GPT-5—fuels the necessity for continuous assessment and adjustment. As AI technology advances, having a clear picture of which models provide the most accurate, dependable solutions will combat productivity pitfalls.
The author reaffirms that developers can’t just expect to use these AI tools blindly and accept what they produce without a critical eye. The analysis of each model’s performance can guide users in selecting the most effective tool for their specific coding tasks.