Anthropic, a prominent player in the artificial intelligence sector, is continuously evolving its hiring process to keep pace with its innovative Claude series of AI models. This year the company debuted a new take-home test for job applicants in 2024, focusing on finding ways to measure applicants’ knowledge and skills. The Claude models, particularly Opus 4 and Opus 4.5, are opening up quickly. This has led to a troubling development — the growing inability to discern between human test-takers and AI-generated responses.
Tristan Hume, technical team lead at Anthropic, explained the nuances that make this challenge so difficult. He explained in a related blog post, posted Wednesday. As he described, that take-home test was put in place exactly for the purpose of testing the skills of people they were looking to hire. As the Claude models evolved, the outcome of such tests became more and more confusing to parse.
The Opus 4 model’s overall proficiency was stunning — it even performed better than the vast majority of human applicants. Just behind in a close second, Opus 4.5 held its own against the top contenders. This level of capability placed Anthropic in a bind. As a result, their AI models consistently produced outputs that were nearly indistinguishable from those of their top human applicants.
“Each new Claude model has forced us to redesign the test,” – Tristan Hume
The ongoing advancement of these foundational AI models means that the take-home test will need to be updated frequently. Hume found her first major challenge. Under the new format, it was impossible to tell the difference between a high-performing job applicant and one of Anthropic’s cutting-edge AI systems.
Since its release, the take-home test has been a central part of Anthropic’s hiring process. The intention here is good—applicants should have a strong grasp of their discipline and the ability to clearly articulate their expertise. What Hume noted is that the development of the Claude models has turned the tables. Now, it’s much more difficult to discern machine produced content from human intelligence.
Even through this external pressure and these challenges, Anthropic is still determined and hopeful about its ability to pivot. Hume said that the company feels uniquely positioned to meet their clients’ changing demands through the hiring process. To this end, they announce an open challenge to anyone who believes they can do better than Opus 4.5 to come beat their team. Apply today for these and other great opportunities at the Agency!
