Over the past few months, developers have raised major alarm bells about these new AI coding assistants. A new analysis by a leading technology executive highlights a troubling trend: while the latest models, GPT-4 and GPT-5, show promise, their performance has raised questions about their reliability. Written by the CEO of Carrington Labs, the author spent several weeks experimenting with different AI systems. He categorized their submissions as constructive, useless, or detrimental. This evaluation offers unique insight on what these tools are doing and how they’re evolving, and what that means for the future of software development.
For the analysis we ran 10 different trials on each model to evaluate their capability to generate useful code. GPT-4 was consistently able to return useful responses, generating a workable solution each time it was put to the test. Previous models such as Claude were often unresponsive when prompted with difficult coding tasks. Meanwhile, GPT-5 emerged as a significant advancement, demonstrating a mix of problem-solving capabilities and occasional shortcomings in adhering to instructions.
Performance Evaluation of GPT-4
Through the initial testing process of GPT-4, we were able to understand its capabilities and limitations. The model did well on all ten attempts and delivered satisfactory responses every single time. In detail, it was charged with incrementing the value of the ‘index_value’ df dataframe column by 1 if that column was available. In 90% of test runs, GPT-4 decided to use a print statement to display the list of columns in the dataframe. The problem was that it always failed to run the command.
Moreover, GPT-4 provided some of the most useful guidance by adding this explanatory comment to its code. The system strongly recommended that users check to see if the requested column exists. If it’s not there, then users need to do something to fix the problem that exists. This proactive approach indicates that while GPT-4 may not execute commands flawlessly, it still provides guidance that can be beneficial for developers navigating coding challenges.
This push for reliability and consistent performance has made GPT-4 a go-to, trusted option for thousands of developers. Its ability to produce valuable information that is provided on a silver platter underscores its tremendous potential for helping with coding tasks. There are significant execution limitations.
The Mixed Results of GPT-5
Unlike its predecessor, GPT-5 performed oddly well and yet so poorly in the exact same code generation task. It hardly ever got it wrong, and when it did, it was by just increasing one for the real index of the row. More troublingly, it displayed a widespread lack of comprehension with the directions provided. In three out of twelve requests, GPT-5 decided that returning only code shouldn’t be followed. Rather, it provided a thorough explanation of why that column was probably not present in the dataset.
GPT-5 attempted to run the command in six-out-of-eight instances. It had exceptions that would either throw an error or fill the new column with an error message if the column didn’t exist. Even so, such behavior shows a much more dynamic approach than seen with GPT-4. It does open up concerns on user clarity and execution consistency.
In one test case, GPT-5 even returned the exact same code as the original, possibly because it was not confident enough to attempt any changes. Even within this performance variation AI coding assistants are fast-evolving. They’re getting better at handling real-world complicated coding situations.
Trends in AI Coding Assistance
The author’s investigation uncovers a disturbing trend. Tasks that remain over five hours with AI help now frequently extend beyond seven or eight hours. With rising time burden on developers, this is the pressing concern. AI coding assistants are failing to achieve the productivity gains we hoped to see from them.
In all, the author submitted these error messages across nine distinct ChatGPT versions—which included mostly iterations of GPT-4 and GPT-5. Yet the results did reveal a key finding. Enhanced models have more advanced abilities and yet still fail at their goal of being a help to coders.
Developers are getting more agitated. They come to the unfortunate conclusion that relying on AI tools will not make them more productive like they expected. We know AI is changing quickly. Innovators should be watchful and flexible as they utilize these innovations in their developments.