Carrington Labs CEO Jamie Twiss recently took a deep dive into AI coding assistants. He notably focused on the performance of GPT-4 and GPT-5 throughout his assessment. In a series of trials, Twiss discovered a concerning trend: while modern models have made strides in code generation, their effectiveness appears to be waning. This article highlights the distinctions between older and newer models and the implications for developers and businesses relying on these technologies.
Thus, Twiss ran ten trials per model to determine which would provide the best supplemental model to generate coding solutions. The results revealed significant differences in performance. In testing, GPT-4 was far more consistently useful with clear answers. The Claude models often had a tough time to even address unresolvable questions appropriately. GPT-5 was our almost-unanimous star performer, reliably churning out the most useful solutions.
A Closer Look at GPT-4
In Twiss’s trials, GPT-4 showed a strong ability to support developers. In fact, it produced functional returns in 90% of the test cases. For example, it would frequently show the names of columns in the returned dataframe and suggest assertions that you verify their existence. Stepping back from error messaging, this proactive process usually led users to fix or debug their code preemptively.
Yet, GPT-4 still had some notable shortcomings. In three of these cases, it did not follow prompts asking for ‘code only’ responses. Rather, it provided more detailed guidance on what to expect, acknowledging that some of the columns may not exist in the dataset. This need to give more background information, while helpful at points, walked away from the brevity of help that so many developers are looking for.
Moreover, GPT-4 encountered challenges when executing code. In six instances, the system attempted to run the code. It started hitting exceptions which either broke it outright or loaded the new columns with erroneous error messages. These limitations make it clear that although GPT-4 can provide useful guidance, it has difficulty generating perfect code every time.
The Rise of GPT-5
By a staggering margin, GPT-5 proved to be uniquely able to provide answers that reliably functioned on every attempt in Twiss’s tests. Our humans just took the real index of each row and incremented it. This philosophy enabled it to consistently and predictably provide the best possible coding solutions. With this kind of reliability, GPT-5 is the most powerful tool yet for developers looking to code more efficiently and effectively.
The step-up from GPT-4 to GPT-5 is impressive. GPT-4 often had difficulty with additional contextual instructions and would misfire on implementation. By comparison, GPT-5 made things easier by focusing on just providing working code. The significance of this performance evolution is that continued progress in AI-based coding assistants can help power even greater productivity boosts for developers.
Even with all this progress, Twiss noticed something pretty unacceptable when he took a look at the AI coding assistant landscape. Core models have hit a quality plateau in 2025, and recent comments point to a degradation of their overall quality. This begs some critical questions about the future of AI coding technologies. Most importantly, can they maintain their competitive advantage amid rapid developer demand?
Implications for Developers and Businesses
Carrington Labs focuses on developing predictive-analytics-based risk models for lenders. In his position, Twiss is almost completely dependent on code produced via LLMs. The disparities between the various AI coding assistants have profound implications for businesses that depend on these tools for efficient software development.
This is important for developers, to illustrate the performance variations that exist within models, even in cases such as GPT-4 vs GPT-5. Understanding what tools are best for what tasks is extremely important. Though GPT-4 remains an effective tool in a wide range of environments, understanding where its limitations lie will help avoid user frustration and misaligned efficiency. Conversely, the high fidelity of GPT-5 provides developers a more reliable alternative.
Businesses need to be on the lookout for iterative changes in AI tech. The potential decline in quality among core models indicates that companies should continuously evaluate their tools and adapt to the evolving landscape. AI-powered coding assistants are rapidly becoming a vital tool for software developers. Now it’s up to organizations to capitalize on the best and most effective solutions out there.
