Perplexity, a tool that provides users with information through an AI-driven interface, recently drew criticism from Cloudflare for its methods of accessing online content. This all reached a fever pitch when Cloudflare took the rare step of publicly denouncing Perplexity, leading to a massive online army of Cloudflare defenders rushing to its defense. Key to this debate is the fact that Perplexity is using a third-party service. Many users and critics believe this has resulted in predatory practices when attempting to use various public-facing websites.
In an extensive Medium blog post, Perplexity fired back at the charge Cloudflare and its CEO, Matthew Prince, had brought against them. Its defense was that what it was doing was just how people access information on the internet. It contended that a legal gray area should not exist between the actions of humans clicking through content online versus those done automatically. Perplexity’s representatives emphasized that the tool’s operation mirrors human behavior, stating, “The difference between automated crawling and user-driven fetching isn’t just technical — it’s about who gets to access information on the open web.”
The firestorm was fanned by the fact that Cloudflare, in their write-up, described examples of Perplexity scraping a specific, publicly available website to answer user questions. This prompted Prince to declare that some “supposedly ‘reputable’ AI companies act more like North Korean hackers,” advocating for the need to “name, shame, and hard block them.” These comments have ignited a Firestorm debate. The key dialogue remains focused on the ethical considerations of AI tools that use web content on behalf of users.
Defenders of Perplexity have not been hard to find on social media, insisting that the service is simply on the right side of the lines. Users have expressed their desire for tools like Perplexity to function effectively, with one supporter stating, “I WANT Perplexity to visit any public content on my behalf when I give it a request/task!” This attitude taps into an expectation from the user community at large, which is that AI should provide a frictionless way to connect with information.
Those opposed to Cloudflare’s stance challenge the distinction drawn between AI tools and regular web browsers. A user on Hacker News posed a critical question: “Why would the LLM accessing the website on my behalf be in a different legal category as my Firefox web browser?” This line of thinking would be true if AI tools worked like ubiquitous, easy-to-use web browsers. It puts into doubt the legitimacy of Cloudflare’s claim to neutrality.
In a blog post, Perplexity further justified their approach by admitting that they use third-party services. They argued that this practice is not necessarily bad. The firm concluded that such services enable searches to be more customized and developed in accordance with ethical boundaries. Cloudflare’s concerns have sparked important discussions about the implications of AI-powered tools for web accessibility and content consumption.
Prince also defended industry standards by pointing to OpenAI, another top company in the AI field. He further pointed out that OpenAI is “taking these best practices to heart.” He further highlighted the organization’s respect for robots.txt files and its dedication to following the rules of the web. This contrast only serves to illuminate the broader ethical milieu in which AI tools are situated. It also highlights the disappointing expectations stakeholders have on the value of these technologies.
Debate continues to rage, but one fact has become apparent: The friction between human web practices and rapidly advancing AI capabilities is a reality. This conflict will be crucial to the future policy landscape. Stakeholders from all sectors are watching this continuing drama very closely. They are particularly interested to learn how it will shape the way AI consumes and interacts with online content.