Recent assessments of AI coding assistants, specifically GPT-4 and its successor GPT-5, reveal a troubling trend: the quality and effectiveness of these models have been declining. In addition to their requirements in data set size and model complexity, both models proved impressive at generating code but were disconcerting to the programming community. This article unpacks our learnings from a grueling series of tests. These tests assessed the usability of these local tools and uncovered major variations in their outputs, underscoring crucial implications for end users.
Across a set of ten different test cases, GPT-4 reliably produced helpful responses, answering every question with success. Upon deeper analysis of its outputs, we found what was in fact an unexpected surprise. In 90% of the instances, GPT-4 just outputted a list of columns from the dataframe. It hardcoded a comment in its code telling users to check if a certain column existed or not. Furthermore, GPT-4 proposed a fix in case the column was absent, indicating an awareness of potential issues but lacking in depth or practical execution.
GPT-5 demonstrated a significantly more sophisticated approach to problem-solving. Second, it was able to find a solution that solved all the time and was able to build novel techniques into its coding. Specifically, one of the steps taken by GPT-5 was to take the index of each row in the dataframe and use that value +1 to define a new column. This approach displayed an impressive grasp of indexing. It confirmed the model’s ability to perform increasingly sophisticated data manipulation tasks.
Performance Comparison: GPT-4 vs. GPT-5
Though each model was successful in its own right, their approaches could not have been more different. Compared to GPT-4, whose often-flippant outputs led some to question its adaptability and, more generally, its usefulness, our model a lot of the time just regurgitates simple code with no real analysis. This limitation can exclude users who demand more advanced solutions.
In six of the test cases, it tried to run the code and included exception catching in its attempt. If the model ever tries to access a required column that isn’t there, it raises an error. Otherwise, it will populate the new column with a user-friendly error message. Such measures evoke a degree of necessary sophistication to more accurately meet the needs of users operating in the landscape of complicated datasets.
Though these developments were promising, GPT-5 was far from perfect. In one case, it even just repeated the code verbatim instead of suggesting an alternative fix. In three cases, GPT-5 failed to follow the instruction to only return code. Instead, it decided to describe why that column was likely missing from the dataset. This new direction to provide rationale instead of just technical products may be daunting to users who are used to plain code generation.
The Implications of Quality Decline
The wider context is that the quality of core AI models— GPT-4 and now GPT-5 —is getting worse. By 2025, these generative models started to hit a quality ceiling. According to recent congressional testimony and reports, their performance is cratering. This trend has serious ramifications for developers and their organizations who are dependent on AI-assisted coding tools.
When these models are clearly not trustworthy, developers are left with an important choice. They need to retest the reliance on these tools for generating code and debugging. The diminishing return on investment in these AI tools could lead to decreased productivity if users cannot trust the outputs provided. As a consequence, organizations may want to spend additional hours on manual coding practices. Or, they can work to find solutions that deliver reliable and verifiable outcomes every time.
Additionally, the reduction in AI performance calls into question the training methodologies used to develop these models. As you know, the world of artificial intelligence is continually changing. It’s incumbent on developers and researchers to improve model performance so we don’t get complacent. Users could use clear improvements in state of the art context comprehension and rectification of contextual errors.