Search Startup Perplexity Increasingly Cites AI-Generated Sources

AI search engine Perplexity claims to be different from other generative AI tools like ChatGPT. Instead of regurgitating data without including any sources, it marks up its short summaries on any topic you want with footnotes that are supposed to link to recent and reliable sources of real-time information drawn from the internet. “Citations are our currency,” CEO Aravind Srinivas told Forbes in April.

But even as the startup has come under fire for republishing the work of journalists without proper attribution, Forbes has learned that Perplexity is also citing as authoritative sources AI-generated blogs that contain inaccurate, out of date and sometimes contradictory information.

According to a study conducted by AI content detection platform GPTZero, Perplexity’s search engine is drawing information from and citing AI-generated posts on a wide variety of topics including travel, sports, food, technology and politics. The study determined if a source was AI-generated by running it through GPTZero’s AI detection software, which provides an estimation of how likely a piece of writing was written with AI with a 97% accuracy rate; for the study, sources were only considered AI-generated if GPTZero determined with at least 95% certainty that they were written with AI (Forbes ran them through an additional AI detection tool called DetectGPT which has a 99% accuracy rate to confirm GPTZero’s assessment).

On average, Perplexity users only need to enter three prompts before they encounter an AI-generated source, according to the study, in which over 100 prompts were tested.

“Perplexity is only as good as its sources,” GPTZero CEO Edward Tian said. “If the sources are AI hallucinations, then the output is too.”

Searches like “cultural festivals in Kyoto, Japan,” “impact of AI on the healthcare industry,” “street food must-tries in Bangkok Thailand,” and “promising young tennis players to watch,” returned answers that cited AI-generated materials. In one example, a search for “cultural festival in Kyoto, Japan” on Perplexity yielded a summary in which the only reference was for an AI-generated LinkedIn post. In another travel-related search for Vietnam’s floating markets, Perplexity’s response, which cited an AI-generated blog, included out-of-date information, the study found.

“Perplexity is only as good as its sources. If the sources are AI hallucinations, then the output is too.”

Perplexity Chief Business Office Dmitri Shevelenko said in an email statement to Forbes that its system is “not flawless” and that it continuously improves its search engine by refining the processes that identify relevant and high quality sources. Perplexity classifies sources as authoritative by assigning “trust scores” to different domains and their content. Its algorithms downrank and exclude websites that contain large amounts of spam, he said. For instance, posts by Microsoft and Databricks are prioritized in search results over others, Shevelenko said.

“As part of this process, we’ve developed our own internal algorithms to detect if content is AI-generated. As with other detectors, these systems are not perfect and need to be continually refined, especially as AI-generated content becomes more sophisticated,” he said.

As AI-generated slop gluts the internet, it becomes more challenging to distinguish between authentic and fake content. And increasingly these synthetic posts are trickling into the products that rely on web sources, bringing with them the inconsistencies or inaccuracies they contain, resulting in “second-hand hallucinations,” Tian said.

“It doesn’t take 50% of the internet being AI to start creating this AI echo chamber,” he told Forbes.

In multiple scenarios, Perplexity relied on AI-generated blog posts, among other seemingly authentic sources, to provide health information. For instance, when Perplexity was prompted to provide “some alternatives to penicillin for treating bacterial infections,” it directly cited an AI-generated blog by a medical clinic that calls itself Penn Medicine Becker ENT & Allergy. (According to GPTZero, it’s 100% likely that the blog is AI-generated. DetectGPT said there is a 94% chance it is fake.)

Such data sources are far from trustworthy because they sometimes offer conflicting information. The AI-generated blog mentions that antibiotics like cephalosporins can be used as an alternative to penicillin for those who are allergic to it, but a few sentences later the post contradicts itself by saying “those with a penicillin allergy should avoid cephalosporins.” Such contradictions were also reflected in answers generated by Perplexity’s AI system, Tian said. The chatbot did, however, suggest consulting a specialist for the safest alternative antibiotic.

^{Got a tip for us? Reach out securely to Rashi Shrivastava at rshrivastava@forbes.com or rashis.17 on Signal.}

Penn Medicine Becker ENT & Allergy customer service representatives redirected Forbes to Penn Medicine. But in response to Forbes’ questions about why the clinic was using AI to generate blogs that gave medical advice, Penn Medicine spokesperson Holly Auer said the specialty physician’s website was not managed by Penn Medicine and that “accuracy and editorial integrity are key standards for all web content associated with our brand, and we will investigate this content and take action as needed.” It’s unclear who manages the website.

Shevelenko said that the study’s examples do not provide “a comprehensive evaluation” of the sources cited by Perplexity but he declined to share data about the types of sources that are cited by the system.

“The reality is that it depends heavily on the types of queries users are asking and their location,” he said. “Someone in Japan asking about the best TV to purchase will yield a very different source set from someone in the U.S. asking about which running shoes to buy.”

Perplexity has also stumbled in its handling of authoritative sources of information. The billion dollar startup recently came under scrutiny for allegations of plagiarizing journalistic work from multiple news outlets including Forbes, CNBC and Bloomberg. Earlier this month, Forbes found Perplexity had lifted sentences, crucial details and custom art from an exclusive Forbes story about Eric Schmidt’s secretive AI drone project without proper attribution. The company recreated the Forbes story across multiple media, in an article, podcast and YouTube video, and pushed it out aggressively to its users with a direct push notification.

“Perplexity represents the inflection point that our AI progress now faces… in the hands of the likes of Srinivas — who has the reputation as being great at the PhD tech stuff and less-than-great at the basic human stuff — amorality poses existential risk,” Forbes Chief Content Officer Randall Lane wrote. Forbes sent a cease and desist letter to Perplexity, accusing the startup of copyright infringement. In response, Perplexity’s CEO Srinivas denied the allegations, arguing that facts cannot be plagiarized, and said that the company has not “‘rewritten,’ ‘redistributed,’ ‘republished,’ or otherwise inappropriately used Forbes content.”

The GPTZero study noted that a Perplexity search for “Eric Schmidt’s AI combat drones,” one of the “pre-recommended” search topics that sits on Perplexity’s landing page, also used a blog post that was written with AI as one of its sources. (GPTZero found that there was a 98% chance the blog was AI-generated while DetectGPT said it was 99% confident.)

“When you use such references, it’s much easier to promote disinformation even if there is no intention to do so.”  

A Wired investigation found that through a secret IP address, the startup had also accessed and scraped work from Wired and other publications owned by media company Condé Nast, even though its engineers had attempted to block Perplexity’s web crawler from stealing content. Even then, the search engine tends to make up inaccurate information and attribute fake quotes to real people. Srinivas did not respond to the Wired story’s claims but said, “The questions from Wired reflect a deep and fundamental misunderstanding of how Perplexity and the Internet work.”

Shevelenko said the company realizes the crucial role that publishers have in creating a healthy information ecosystem that its product depends on. To that end, Perplexity has created what it claims is a “first-of-its-kind” revenue sharing program that will compensate publishers in a limited capacity. It plans to add an advertising layer on its platform that will allow brands to sponsor follow-up or “related” questions in its search and Pages products. For specific responses generated by its AI where Perplexity earns revenue, the publishers that are cited as a source in that answer will receive a cut. The company did not share what percentage of revenue it plans to share. It has been in talks with The Atlantic among other publishers about potential partnerships.

Srinivas, who was a researcher at OpenAI before starting Perplexity in 2022, has raised over $170 million in venture funding (per Pitchbook). The company’s backers include some of the most high-profile names in tech, including Amazon founder Jeff Bezos, Google Chief Scientist Jeff Dean, former YouTube CEO Susan Wojcicki, Open AI cofounder Andrej Karpathy and Meta Chief Scientist Yann LeCun. In recent months, its conversational search chatbot has exploded in popularity, with 15 million users that include billionaires like Nvidia CEO Jensen Huang and Dell founder and CEO Michael Dell.

Perplexity uses a process called “RAG” or retrieval-augmented generation, which allows an AI system to retrieve real time information from external data sources to improve its chatbot’s responses. But a degradation in the quality of these sources could have a direct impact on the responses its AI produces, experts say.

Zak Shumaylov, a machine learning researcher at the University of Cambridge, said if real time sources themselves contain biases or inaccuracies, any application built on top of such data could eventually experience a phenomenon called model collapse, where an AI model that is trained on AI-generated data starts “spewing nonsense because there is no longer information, there is only bias.”

“When you use such references, it’s much easier to promote disinformation even if there is no intention to do so,” he said.

Relying on low-quality web sources is a widespread challenge for AI companies, many of which don’t cite sources at all. In May, Google’s “AI overviews,” a feature that uses AI to generate previews on a topic, produced an array of misleading responses like suggesting adding glue to stick cheese on pizza and claiming that eating rocks can be good for your health. Part of the problem was that the system appeared to be pulling from unvetted sources like discussion forums on Reddit and satirical sites like The Onion. Liz Reid, head of Google Search, admitted in a blog that some erroneous results appeared on Google in part because of a lack of quality information on certain topics.

“Perplexity is only one case,” Tian said. “It’s a symptom, not the entire problem.”

MORE FROM FORBES

ForbesBuzzy AI Search Engine Perplexity Is Directly Ripping Off Content From News OutletsBy Sarah EmersonForbes‘I Want That Sweet Baby’: AI-Generated Kids Draw Predators On TikTok And InstagramBy Alexandra S. LevineForbesVinod Khosla, Marc Andreessen And The Billionaire Battle For AI’s FutureBy Alex KonradForbesHow Stability AI’s Founder Tanked His Billion-Dollar StartupBy Kenrick Cai