For us at OpenAI, it’s been an amazing and transformative ride. We’re building AI technologies in the most difficult way possible, developing solutions that will allow systems to act on complicated tasks just as humans would. As part of this endeavor, the organization recently recruited five leading researchers from Meta’s new superintelligence-focused unit, offering compensation packages exceeding $100 million. This relatively simple yet strategic move is a clear signal of OpenAI’s commitment to advancing more compelling research. It’s positioning the company as an AI innovation leader, too.
Advancing AI to deepen reasoning capabilities has been one of OpenAI’s top priorities recently. The original team, known as the “Agents” team, to build reasoning enabled systems that could accomplish tasks through logical thought processes. Yet, from the outset, the distinction between reasoning models vs agents wasn’t clear cut. OpenAI’s contributions have been cumulative and their progress impressive. Its unique IMO model creates hundreds of agents that pursue hundreds of ideas at once, allowing for greater diversity and experimentation to find the winning solution. This strategy builds toward alternative structures and uses of AI.
OpenAI’s path took off in earnest with the release of its first large language model (LLM) in 2018. This model was pretrained on massive amounts of internet data. It did so by using a massive cluster of GPUs to fuel further innovations. From left to right, in 2022, Hani Farah and Hunter Lightman at OpenAI. He’s seen, first-hand, the explosive growth of ChatGPT, the quickest-growing product in history. Creating the o1 reasoning model required OpenAI to reallocate vital resources, particularly talent and GPUs, which posed challenges for ongoing projects.
The Evolution of AI Reasoning Models
OpenAI’s research into AI reasoning has been all the rage in recent years—sparked by major breakthroughs in 2023. The smart system that would manage this was first dubbed “Q*,” but eventually rebranded as “Strawberry.” It extends LLMs with reinforcement learning (RL) and an approach called test-time computation. Together, these technologies hold tremendous potential to improve the way AI systems can better reason and carry out tasks.
Daniel Selsam heads up OpenAI’s Agents team, which has been at the forefront of developing this exciting new paradigm. In particular, the team is making strides in improving AI’s ability to simulate human-like reasoning processes. As El Kishky, a researcher at OpenAI, explained it, “I could tell the model was beginning to reason. It would misspelled words and retrace its steps. It would lose patience. It was just like being inside a character’s head.” This testimony focuses on the definition of AI systems that are most likely to be capable of problem-solving in ways that align with human thinking.
Even with these developments, OpenAI admits that its models still have a long way to go. Even the new ChatGPT-4 systems continue to show proclivities to hallucinate and have difficulties with multi-step tasks. Lightman stressed that it really comes down to the data. As with most machine learning issues, it’s a data issue,” he added, emphasizing the importance of quality and quantity of training data in improving AI’s performance.
The Impact of Collaboration and Resource Allocation
OpenAI’s success to date has been driven in large part by cooperation between researchers and coordinated use of resources. Frequent negotiations with company leaders to secure the resources needed to make it all work is par for the course inside the ship. It’s a simple fact that showing concrete advances is a successful motivating tactic for academics trying to win more research funding.
Lightman shared insights into this process: “When we showed the evidence [for o1], the company was like, ‘This makes sense, let’s push on it.’” This partnership model fosters innovation by making sure bold, promising projects get the attention and resources they need to make it through to development. Researchers function within a bottom-up framework, which creates fertile ground for such bold and innovative ideas to flourish and grow organically from within an interdisciplinary team.
The Codex agent is a perfect example of OpenAI’s initiative to help software engineers by using AI to complete simple coding tasks. This initiative is part of a larger effort to make AI tools more effective and easier to use. The team’s next phase of research will build upon this work to extend these agents’ abilities even further, particularly with respect to mathematical reasoning. In particular, Lightman noted, “I think these models are going to be really good at math. They’re going to get better at this kind of reasoning too.”
Future Aspirations and Challenges Ahead
Looking forward Further down the line, OpenAI imagines a world in which users can simply ask AI systems to accomplish large tasks that they’ll complete on their own. Sam Altman, co-founder of OpenAI, expressed this aspiration succinctly: “Eventually, you’ll just ask the computer for what you need and it’ll do all of these tasks for you.” Embedded within this vision is OpenAI’s desire to ensure that AI is useful and valuable to the average user.
Open AI is certainly laudable in the lofty goals it has laid out, producing bona fide superintelligence is still no small feat. The organization’s leadership is deeply committed to addressing the shortcomings of its existing models. Simultaneously, it pushes for improvements that will improve deep reasoning skills. Lightman shared that when the model is asked to perform hard tasks, it begins to change the way it reasons. It builds out whatever magic wizardry is needed to win. This view of approximation as a process highlights the importance of continued research and development.